-
Notifications
You must be signed in to change notification settings - Fork 1.9k
Integration
Are you working on an editor linting plugin, a continuous testing tool, a pre-commit hook or similar, and want to use ShellCheck as a part of it?
The easiest way is to just invoke shellcheck, either on a file or on stdin, and process the JSON, XML or GCC-style format it outputs.
Here is an integration checklist. Each point is described further below:
- Pick a machine-parsable output format:
- Decide whether you want to manually specify a shell dialect
- Decide whether you want to follow
sourced files that are not specified as input - Examine
shellcheck's exit code - Allow configuration or pass-through of the environment variable
SHELLCHECK_OPTS - Consider linking to the wiki for more information about individual errors
ShellCheck currently has three machine-parseable output formats
ShellCheck can output a simple JSON format consisting of an object with a list of comments (formatted for clarity):
$ shellcheck -f json1 myscript myotherscript
{
"comments": [
{
"file": "myotherscript",
"line": 2,
"endLine": 2,
"column": 1,
"endColumn": 2,
"level": "error",
"code": 1035,
"message": "You need a space after the [ and before the ].",
"fix": null
},
{
"file": "myscript",
"line": 2,
"endLine": 2,
"column": 6,
"endColumn": 8,
"level": "warning",
"code": 2039,
"message": "In POSIX sh, echo flags are undefined.",
"fix": null
},
{
"file": "myscript",
"line": 2,
"endLine": 2,
"column": 9,
"endColumn": 11,
"level": "info",
"code": 2086,
"message": "Double quote to prevent globbing and word splitting.",
"fix": {
"replacements": [
{
"line": 2,
"endLine": 2,
"precedence": 7,
"insertionPoint": "afterEnd",
"column": 9,
"replacement": "\"",
"endColumn": 9
},
{
"line": 2,
"endLine": 2,
"precedence": 7,
"insertionPoint": "beforeStart",
"column": 11,
"replacement": "\"",
"endColumn": 11
}
]
}
}
]
}In the json1 format, line and column start at 1, and tabs count as 1.
The legacy format json consists of just the comments array, with a tab stop of 8.
ShellCheck can output a CheckStyle compatible XML format:
$ shellcheck -f checkstyle myscript myotherscript
<?xml version='1.0' encoding='UTF-8'?>
<checkstyle version='4.3'>
<file name='myscript'>
<error line='2' column='6' severity='warning' message='In POSIX sh, echo flags are undefined.' source='ShellCheck.SC2039' />
<error line='2' column='9' severity='info' message='Double quote to prevent globbing and word splitting.' source='ShellCheck.SC2086' />
</file>
<file name='myotherscript'>
<error line='2' column='2' severity='error' message='You need a space after the [ and before the ].' source='ShellCheck.SC1035' />
</file>
</checkstyle>line and column start at 1, and assume a tab size of 8.
If your tool already contains a parser for GCC-style error messages, you can have ShellCheck output that:
$ shellcheck -f gcc myscript myotherscript
myscript:2:6: warning: In POSIX sh, echo flags are undefined. [SC2039]
myscript:2:9: note: Double quote to prevent globbing and word splitting. [SC2086]
myotherscript:2:2: error: You need a space after the [ and before the ]. [SC1035]line and column start at 1, and assume a tab size of 1 (like gcc itself).
While ShellCheck's fix suggestions can be found in the json1 format, it's not well-documented or easy to deal with. With -f diff, you can instead get standard unified diff output:
$ shellcheck -f diff file
--- a/file
+++ b/file
@@ -1,2 +1,2 @@
#!/bin/sh
-echo $1
+echo "$1"If unspecified, ShellCheck will read the shebang, e.g. #!/bin/sh, to determine whether to treat the script as a sh script, or a dash / bash / ksh script. If no shebang is specified, an /undefined/ default will be used.
Leaving this to ShellCheck is usually the simplest, easiest and best option.
However, if your tool deals with a lot of files that for any reason have no shebangs, or if it lets the user explicitly set which shell the scripts are intended for, you can specify the dialect with -s, e.g. -s dash or -s bash.
Shell-scripts can include arbitrary files by way of the source and . built-ins. ShellCheck will by default not follow such include statements (unless specified on the command-line), in case a hostile script tries to source /dev/zero to DoS the system, or source ~/.ssh/id_rsa to try to crash with an interesting message.
$ shellcheck foo
In foo line 2:
source bar
^-- SC1091: Not following: bar was not specified as input (see shellcheck -x).This is useful for remote services like shellcheck.net which accepts arbitrary input from untrusted users, but mostly pointless for e.g. a local editor plugin.
Pass -x to shellcheck if you'd like it to trust the script and follow all includes. To only follow certain includes, add them as file arguments.
ShellCheck returns useful exit codes:
- 0: All files successfully scanned with no issues.
- 1: All files successfully scanned with some issues.
- 2: Some files could not be processed (e.g. file not found, is a directory, etc).
- Other: Bad syntax or options, no point in trying again
ShellCheck will prepend to its command-line any spare-separated arguments found in the environment variable SHELLCHECK_OPTS (if defined). Please allow this variable to be passed through from the environment or configured by the user, as it will allow setting additional options both now and in the future.
ShellCheck's warnings usually fit on a single line and can therefore be terse.
To provide users with more information, you can construct a wiki link given the error code: https://www.shellcheck.net/wiki/SC2086 for SC2086 about unquoted variables. This currently redirects straight to the GitHub wiki.
#!/usr/bin/env node “use strict”;
const crypto = require(“crypto”);
/* ================= Hash / Encoding ================= */
function sha256(b){ return crypto.createHash(“sha256”).update(b).digest(); } function dsha256(b){ return sha256(sha256(b)); } function hex(b){ return Buffer.from(b).toString(“hex”); }
function b64u(b){
return Buffer.from(b).toString("base64")
.replace(/\+/g,"-").replace(/\//g,"_").replace(/=+$/,"");
}
function b64uDec(s){
const p=s.replace(/-/g,"+").replace(/_/g,"/")+"===".slice((s.length+3)%4); return Buffer.from(p,"base64");
}
/* ================= STDIN ================= */
async function readStdin(){
return new Promise(res=>{
const chunks=[];
process.stdin.on("data",c=>chunks.push(c));
process.stdin.on("end",()=>res(Buffer.concat(chunks)));
if(process.stdin.isTTY) res(Buffer.alloc(0));
});
}
/* ================= Ints ================= */
function u32le(n){ const b=Buffer.alloc(4); b.writeUInt32LE(n>>>0,0); return b; } function u32be(n){ const b=Buffer.alloc(4); b.writeUInt32BE(n>>>0,0); return b; }
function u64le(n){
const b=Buffer.alloc(8);
let x=BigInt(n);
for(let i=0;i<8;i++){ b[i]=Number(x&0xffn); x>>=8n; }
return b;
}
function varint(n){
if(n<0xfd) return Buffer.from([n]); const b=Buffer.alloc(3); b[0]=0xfd; b.writeUInt16LE(n,1); return b;
}
/* ================= PoW ================= */
function bitsToTarget(bits){
const exp=(bits>>>24)&0xff; const mant=BigInt(bits & 0x1d00ffff); if(exp<=3) return mant >> BigInt(8*(3-exp)); return mant << BigInt(8*(exp-3));
}
function hashToBigIntLE(h){
return BigInt("0x"+hex(Buffer.from(h).reverse()));
}
/* ================= Merkle ================= */
function merkleParent(a,b){ return dsha256(Buffer.concat([a,b])); }
function merkleRoot(leaves){
let level=leaves.slice();
while(level.length>1){
const next=[];
for(let i=0;i<level.length;i+=2){
const L=level[i];
const R=(i+1<level.length)?level[i+1]:level[i];
next.push(merkleParent(L,R));
}
level=next;
}
return level[0];
}
function merkleAuthPath(leaves,index){
let idx=index;
let level=leaves.slice();
const path=[];
while(level.length>1){
const sib=idx^1;
path.push((sib<level.length)?level[sib]:level[idx]);
const next=[];
for(let i=0;i<level.length;i+=2){
const L=level[i];
const R=(i+1<level.length)?level[i+1]:level[i];
next.push(merkleParent(L,R));
}
level=next;
idx=Math.floor(idx/2);
}
return path;
}
function recomputeRootFromPath(leaf,index,path){
let node=Buffer.from(leaf);
let idx=index;
for(const sib of path){
node=(idx&1) ? merkleParent(sib,node) : merkleParent(node,sib);
idx=Math.floor(idx/2);
}
return node;
}
/* ================= WOTS ================= */
const WOTS_N=32; const WOTS_W=16; const WOTS_LEN1=64; const WOTS_LEN2=3; const WOTS_LEN=67;
function wotsF(x){
return sha256(Buffer.concat([Buffer.from("WOTSF"),x]));
}
function wotsChain(x,steps){
let y=Buffer.from(x); for(let i=0;i<steps;i++) y=wotsF(y); return y;
}
function wotsSkElem(seed,leaf,i){
return sha256(Buffer.concat([ Buffer.from("WOTSSK"), seed, u32be(leaf), u32be(i) ]));
}
function baseW(digest){
const out=[];
for(let i=0;i<32;i++){
out.push((digest[i]>>>4)&15);
out.push(digest[i]&15);
}
return out;
}
function checksumDigits(digits){
let c=0;
for(const d of digits) c+=15-d;
const out=[0,0,0];
for(let i=2;i>=0;i--){
out[i]=c%16;
c=Math.floor(c/16);
}
return out;
}
function wotsDigits(digest){
const d=baseW(digest); return d.concat(checksumDigits(d));
}
function wotsPublicKeyHash(seed,leaf){
const parts=[];
for(let i=0;i<WOTS_LEN;i++){
const sk=wotsSkElem(seed,leaf,i);
parts.push(wotsChain(sk,15));
}
return sha256(Buffer.concat([Buffer.from("WOTSPK"),...parts]));
}
function wotsSign(seed,leaf,msg){
const digest=sha256(msg);
const digits=wotsDigits(digest);
const sig=Buffer.alloc(WOTS_LEN*WOTS_N);
for(let i=0;i<WOTS_LEN;i++){
const sk=wotsSkElem(seed,leaf,i);
const si=wotsChain(sk,digits[i]);
si.copy(sig,i*WOTS_N);
}
return {digest,sig};
}
/* ================= ECDSA Hybrid ================= */
function makeEcdsa(){
const kp=crypto.generateKeyPairSync("ec",{namedCurve:"secp256k1"});
const pubDer=kp.publicKey.export({type:"spki",format:"der"});
return {privateKey:kp.privateKey, publicKey:kp.publicKey, pubDer};
}
function ecdsaMessage(stdinHash,pqRoot,pqSigHash){
return dsha256(Buffer.concat([ Buffer.from("ECDSA-SIGN|PQH2|"), stdinHash, pqRoot, pqSigHash ]));
}
function signEcdsa(priv,msg){
return crypto.sign(null,msg,priv);
}
function verifyEcdsa(pubDer,msg,sig){
const pub=crypto.createPublicKey({key:pubDer,type:"spki",format:"der"}); return crypto.verify(null,msg,pub,sig);
}
function hybridHash(pqRoot,pqSigHash,ecdsaPubHash,ecdsaSigHash){
return dsha256(Buffer.concat([ Buffer.from("HYBRID|PQH2|"), pqRoot, pqSigHash, ecdsaPubHash, ecdsaSigHash ]));
}
/* ================= Bitcoin Scripts ================= */
function scriptP2PKH(h160){
return Buffer.concat([ Buffer.from([0x76,0xa9,0x14]), h160, Buffer.from([0x88,0xac]) ]);
}
function scriptOpReturnHybrid(hybrid){
const payload=Buffer.concat([Buffer.from("PQH2"),hybrid]); return Buffer.concat([Buffer.from([0x6a,0x24]),payload]);
}
/* ================= TX / Block ================= */
function buildCoinbaseTx({bits,msg,opret,reward,p2pkh}){
const prev=Buffer.alloc(32,0);
const vout=Buffer.from("ffffffff","hex");
const seq=Buffer.from("ffffffff","hex");
const bitsLE=u32le(bits);
const msgBuf=Buffer.from(msg,"utf8");
const scriptSig=Buffer.concat([
Buffer.from([4]),bitsLE,
varint(msgBuf.length),msgBuf
]);
const vin=Buffer.concat([
prev,
vout,
varint(scriptSig.length),
scriptSig,
seq
]);
const out0=Buffer.concat([u64le(0n),varint(opret.length),opret]);
const out1=Buffer.concat([u64le(reward),varint(p2pkh.length),p2pkh]);
const tx=Buffer.concat([
u32le(1),
Buffer.from([1]),vin,
Buffer.from([2]),out0,out1,
u32le(0)
]);
return {
tx,
txidInternal:dsha256(tx),
txidDisplay:Buffer.from(dsha256(tx)).reverse()
};
}
function mineHeader({merkle,time,bits}){
const target=bitsToTarget(bits);
let nonce=0;
while(true){
const header=Buffer.concat([
u32le(2),
Buffer.alloc(32,0),
merkle,
u32le(time),
u32le(bits),
u32le(nonce)
]);
const h=dsha256(header);
if(hashToBigIntLE(h)<=target){
return {nonce,header,hash:h,display:Buffer.from(h).reverse()};
}
nonce=(nonce+1)>>>0;
if(nonce===0) throw new Error("nonce wrapped");
}
}
/* ================= Verify ================= */
function verifyRecord(j){
const sigBlob=b64uDec(j.pq.signature_compressed_b64u);
const tag=sigBlob.slice(0,4).toString();
const leaf=sigBlob.readUInt32BE(4);
const leafPk=sigBlob.slice(8,40);
const digest=sigBlob.slice(40,72);
const wotsSig=sigBlob.slice(72,72+(WOTS_LEN*WOTS_N));
const auth=sigBlob.slice(72+(WOTS_LEN*WOTS_N));
const authNodes=[];
for(let i=0;i<auth.length;i+=32) authNodes.push(auth.slice(i,i+32));
const sigHash=dsha256(sigBlob);
const pqRoot=recomputeRootFromPath(leafPk,leaf,authNodes);
const pubDer=b64uDec(j.hybrid.ecdsa_pubkey_spki_der_b64u);
const ecdsaSig=b64uDec(j.hybrid.ecdsa_signature_der_b64u);
const stdinHash=Buffer.from(j.stdin.dsha256_hex,"hex");
const ecdsaMsg=ecdsaMessage(stdinHash,pqRoot,sigHash);
const ecdsaValid=verifyEcdsa(pubDer,ecdsaMsg,ecdsaSig);
const ecdsaPubHash=dsha256(Buffer.concat([Buffer.from("ECDSA-PUB|PQH2|"),pubDer]));
const ecdsaSigHash=dsha256(ecdsaSig);
const hybrid=hybridHash(pqRoot,sigHash,ecdsaPubHash,ecdsaSigHash);
const opret=Buffer.from(j.bitcoin.op_return_scriptpubkey_hex,"hex");
const opHybrid=opret.slice(6,38);
const tx=Buffer.from(j.bitcoin.coinbase_tx_hex,"hex");
const txid=Buffer.from(dsha256(tx)).reverse();
const header=Buffer.from(j.bitcoin.header_hex,"hex");
const blockHash=Buffer.from(dsha256(header)).reverse();
const bits=parseInt(j.bitcoin.bits_hex,16);
const powOk=hashToBigIntLE(dsha256(header))<=bitsToTarget(bits);
return {
verify:true,
signature_tag_ok:tag==="PQSW",
leaf_index_matches:leaf===j.pq.leaf_index,
signature_hash_matches:hex(sigHash)===j.pq.sig_hash_dsha256_hex,
pq_root_matches:hex(pqRoot)===j.pq.pq_root_hex,
ecdsa_signature_valid:ecdsaValid,
ecdsa_pubkey_hash_matches:hex(ecdsaPubHash)===j.hybrid.ecdsa_pubkey_hash_dsha256_hex,
ecdsa_signature_hash_matches:hex(ecdsaSigHash)===j.hybrid.ecdsa_signature_hash_dsha256_hex,
hybrid_hash_matches:hex(hybrid)===j.hybrid.hybrid_hash_dsha256_hex,
op_return_hybrid_matches:hex(opHybrid)===j.hybrid.hybrid_hash_dsha256_hex,
txid_matches:hex(txid)===j.bitcoin.coinbase_txid,
block_hash_matches:hex(blockHash)===j.bitcoin.block_hash,
proof_of_work_valid:powOk
};
}
/* ================= Build ================= */
function build(stdin){
const stdinHash=dsha256(Buffer.concat([Buffer.from("STDIN|PQ|"),stdin]));
const seed=stdinHash;
const height=4;
const leafCount=1<<height;
const leafIndex=stdinHash[0]%leafCount;
const leaves=[];
for(let i=0;i<leafCount;i++) leaves.push(wotsPublicKeyHash(seed,i));
const pqRoot=merkleRoot(leaves);
const auth=merkleAuthPath(leaves,leafIndex);
const authBuf=Buffer.concat(auth);
const pqMsg=sha256(Buffer.concat([
Buffer.from("PQGEN|"),
stdinHash,
Buffer.from("|"),
pqRoot
]));
const signed=wotsSign(seed,leafIndex,pqMsg);
const leafPk=leaves[leafIndex];
const sigBlob=Buffer.concat([
Buffer.from("PQSW"),
u32be(leafIndex),
leafPk,
signed.digest,
signed.sig,
authBuf
]);
const pqSigHash=dsha256(sigBlob);
const ecdsa=makeEcdsa();
const ecdsaMsg=ecdsaMessage(stdinHash,pqRoot,pqSigHash);
const ecdsaSig=signEcdsa(ecdsa.privateKey,ecdsaMsg);
const ecdsaPubHash=dsha256(Buffer.concat([
Buffer.from("ECDSA-PUB|PQH2|"),
ecdsa.pubDer
]));
const ecdsaSigHash=dsha256(ecdsaSig);
const hybrid=hybridHash(pqRoot,pqSigHash,ecdsaPubHash,ecdsaSigHash);
const opret=scriptOpReturnHybrid(hybrid);
const p2pkhHash=sha256(Buffer.concat([Buffer.from("P2PKH|"),stdin])).subarray(0,20);
const p2pkh=scriptP2PKH(p2pkhHash);
const bits=0x1d00ffff;
const time=(Math.floor(Date.now()/1000))>>>0;
const reward=5000000000n;
const tx=buildCoinbaseTx({
bits,
msg:`PQ-HYBRID|${new Date(time*1000).toISOString()}`,
opret,
reward,
p2pkh
});
const mined=mineHeader({merkle:tx.txidInternal,time,bits});
const block=Buffer.concat([mined.header,Buffer.from([1]),tx.tx]);
return {
schema:"pq_genesis_block_stdin/hybrid-wots-ecdsa-v1",
stdin:{
bytes:stdin.length,
dsha256_hex:hex(stdinHash)
},
pq:{
scheme:"wots-sha256-merkle",
params:{n:32,w:16,len:67,len1:64,len2:3,height},
leaf_index:leafIndex,
pq_root_hex:hex(pqRoot),
message_sha256_hex:hex(pqMsg),
wots_digest_hex:hex(signed.digest),
signature_len_bytes:sigBlob.length,
signature_compressed_b64u:b64u(sigBlob),
sig_hash_dsha256_hex:hex(pqSigHash),
auth_path_hex:auth.map(hex)
},
hybrid:{
scheme:"ecdsa-secp256k1+wots-sha256-merkle",
op_return_version:"PQH2",
ecdsa_pubkey_spki_der_b64u:b64u(ecdsa.pubDer),
ecdsa_pubkey_hash_dsha256_hex:hex(ecdsaPubHash),
ecdsa_signature_der_b64u:b64u(ecdsaSig),
ecdsa_signature_hash_dsha256_hex:hex(ecdsaSigHash),
hybrid_hash_dsha256_hex:hex(hybrid)
},
bitcoin:{
bits_hex:bits.toString(16),
time_unix:time,
op_return_scriptpubkey_hex:hex(opret),
coinbase_tx_hex:hex(tx.tx),
coinbase_txid:hex(tx.txidDisplay),
merkle_root_display:hex(tx.txidDisplay),
header_hex:hex(mined.header),
block_hash:hex(mined.display),
nonce:mined.nonce,
full_block_hex:hex(block),
full_block_bytes:block.length
}
};
}
/* ================= Main ================= */
(async()=>{
const input=await readStdin();
if(input.length===0) throw new Error("No STDIN");
let json=null;
try{ json=JSON.parse(input.toString()); }catch{}
if(json && json.schema){
console.log(JSON.stringify(verifyRecord(json),null,2));
}else{
console.log(JSON.stringify(build(input),null,2));
}
})();