¶ Summary
The objective is to monitor market sentiment on agricultural commodity futures (bullish/bearish) by extracting key metrics from the weekly COT report published by the US CFTC (link). The challenge lies in setting up a robust, long-term process that remains reliable despite minor, unforeseen changes in the report’s format.
The report’s old-style typographic layout and lack of formal format specifications present obstacles. A solution must be developed that can either infer the structure through detailed analysis or handle the variability with a resilient extraction mechanism.
Two approaches are evaluated:
(1.) A custom lexer, developed with deep regex expertise and a thorough text analysis, designed to tokenize the input based on deterministic rules. This lexer produces highly regular token patterns, simplifying downstream tagging.
(2.) A generic tokenizer, requiring minimal prior knowledge and no regex development. Although more flexible, this method generates less predictable token sequences, likely requiring machine-learning-based Sequence labeling (e.g., CRF++).
Both approaches are assessed for strengths and limitations, focusing on their resilience to input variability and suitability for long-term automation. The evaluation informs the choice of extraction strategy to ensure consistent, accurate sentiment monitoring.
¶ Table of Content
building the tech stack
data source
goal
input
preliminary analysis
Lexer program
output
the machine-learning approach
¶ Project execution
¶ building the tech stack
Below is the supply-demand-solution diagram that shows how the tools we have shortlisted can be combined to meet the requirements. Both reasons for adoption and reasons for dismissal are noted.
¶ data source
weekly report COT by CFTC : https://www.cftc.gov/dea/options/ag_lof.htm - cached
Disaggregated Futures-and-Options -Combined / May 02, 2023 / Agriculture / Long Format
¶ goal
to extract, for each product,
- the product label (line 1),
- the product code (line 1),
- the report date (line 2),
- the long and short Positions held by Managed Money traders, All crop years (line 11),
- the Changes of the two figures above w.r.t the previous week (line 16)
and to eventually show, for each product,
- the current Bull/Bear Ratio
- the previous Bull/Bear Ratio
¶ input
WHEAT-SRW - CHICAGO BOARD OF TRADE Code-001602
Disaggregated Commitments of Traders - Options and Futures Combined, May 02, 2023
-------------------------------------------------------------------------------------------------------------------------------------------------------------
: : Reportable Positions : Nonreportable
: : Producer/Merchant/ : : : : Positions
: Open : Processor/User : Swap Dealers : Managed Money : Other Reportables :
: Interest : Long : Short : Long : Short :Spreading : Long : Short :Spreading : Long : Short :Spreading : Long : Short
-------------------------------------------------------------------------------------------------------------------------------------------------------------
: :(CONTRACTS OF 5,000 BUSHELS) :
: : Positions :
All : 462,482: 78,394 51,330 71,087 4,461 14,141 58,776 185,100 70,898 37,166 13,208 86,792: 45,229 36,552
Old : 457,133: 77,419 49,598 70,363 4,321 13,838 58,693 184,417 70,506 37,592 14,426 84,476: 44,246 35,551
Other: 5,349: 975 1,732 940 356 86 475 1,075 0 1,397 606 492: 983 1,001
: : :
: : Changes in Commitments from: April 25, 2023 :
: 32,903: 6,760 -4,310 158 918 559 551 13,862 5,039 4,462 966 16,542: -1,169 -675
: : :
: : Percent of Open Interest Represented by Each Category of Trader :
All : 100.0: 17.0 11.1 15.4 1.0 3.1 12.7 40.0 15.3 8.0 2.9 18.8: 9.8 7.9
Old : 100.0: 16.9 10.8 15.4 0.9 3.0 12.8 40.3 15.4 8.2 3.2 18.5: 9.7 7.8
Other: 100.0: 18.2 32.4 17.6 6.7 1.6 8.9 20.1 0.0 26.1 11.3 9.2: 18.4 18.7
: : :
: : Number of Traders in Each Category :
All : 376: 80 69 24 5 21 53 81 70 70 41 63:
Old : 376: 80 68 24 5 21 53 81 70 72 40 63:
Other: 75: 13 25 10 5 . 5 6 0 4 11 10:
:-------------------------------------------------------------------------------------------------------------------------------------------------------
: Percent of Open Interest Held by the Indicated Number of the Largest Traders
: By Gross Position By Net Position
: 4 or Less Traders 8 or Less Traders 4 or Less Traders 8 or Less Traders
: Long: Short Long Short: Long Short Long Short
:----------------------------------------------------------------------------------------------------
All : 12.0 12.6 20.4 21.8 8.8 9.7 14.6 16.9
Old : 12.1 12.6 20.5 21.9 8.9 9.8 14.7 17.1
Other: 40.0 49.6 53.6 60.4 37.8 44.9 49.9 54.5
WHEAT-HRW - CHICAGO BOARD OF TRADE Code-001612
Disaggregated Commitments of Traders - Options and Futures Combined, May 02, 2023
-------------------------------------------------------------------------------------------------------------------------------------------------------------
: : Reportable Positions : Nonreportable
: : Producer/Merchant/ : : : : Positions
: Open : Processor/User : Swap Dealers : Managed Money : Other Reportables :
: Interest : Long : Short : Long : Short :Spreading : Long : Short :Spreading : Long : Short :Spreading : Long : Short
..............
¶ preliminary analysis
given that we are analyzing a raw text file (not csv, not json, etc.),
gawk and a 200-char script can already provide valuable information on the file structure and uncover any high-level recurring pattern:
# gawk script
BEGIN {X=0; m=1; tab="\t"; print "SN" tab "NR" tab "NRgap" tab "input line"}
X==0 && /^[[:space:]]*$/ { X=1; next }
X==1 && /[[:space:]]+Code-[[:alnum:]]+$/ {
print m++ tab NR tab NR-(NRprev?NRprev:NR) tab $0
NRprev=NR; X=0; next
}
the simple parsing above shows that there are 25 product sections and sections are regularly 36-line long.
SN NRin NRgap input line
1 2 0 WHEAT-SRW - CHICAGO BOARD OF TRADE Code-001602
2 38 36 WHEAT-HRW - CHICAGO BOARD OF TRADE Code-001612
3 74 36 WHEAT-HRSpring - MINNEAPOLIS GRAIN EXCHANGE Code-001626
4 110 36 CORN - CHICAGO BOARD OF TRADE Code-002602
5 146 36 CORN CONSECUTIVE CSO - CHICAGO BOARD OF TRADE Code-00260B
6 182 36 OATS - CHICAGO BOARD OF TRADE Code-004603
7 218 36 ROUGH RICE - CHICAGO BOARD OF TRADE Code-039601
...
22 758 36 FRZN CONCENTRATED ORANGE JUICE - ICE FUTURES U.S. Code-040701
23 794 36 COCOA - ICE FUTURES U.S. Code-073732
24 830 36 SUGAR NO. 11 - ICE FUTURES U.S. Code-080732
25 866 36 COFFEE C - ICE FUTURES U.S. Code-083731
¶ Lexer program
A lexer and a tokenizer perform the same task.
In this page,
we call tokenizer a sofware that gets instructed with definitions of inter-token delimiters (usually spaces or punctuation marks),
and we call lexer a software that accepts definitions of tokens - often through REGEXes).
Usually, a tokenizer never terminates with a fatal exception (edge case: an only token is returned) while a lexer may do.
A pure Lexer is a set of rules by which a sequence of characters is identified and labeled, independently of context. For example, given the input "380 Airbus A380" the Lexer and its rules may return: NUMBER(380) WORD(Airbus) WORD(A380).
A byte-pair sequence analysis based on utilities such as GNU od ("object dump") leads to the following finite-state-machine, that we will keep in mind while developing the ad-hoc, regex-based Lexer:
The Lexer developed in the Groovy language is as follows:
import java.util.regex.Pattern
import java.util.regex.Matcher
assert args.size()==1
String fileContents = new File(args[0]).getText('UTF-8')
List get_matches(String text,def re, String groupnames_csv) {
List groupnames = groupnames_csv.tokenize(",")
List ret = []
Pattern pattern = Pattern.compile(re,Pattern.MULTILINE)
Matcher matcher = pattern.matcher(text)
while (matcher.find()) {
groupnames.each { grp_nm ->
String s = matcher.group(grp_nm)
if(s) ret << [ a:matcher.start(), z:matcher.end(), sz:s.size(), grp:grp_nm, lines:s.tokenize("\n").size(), s:s ]
}
}
return ret
}
skipFields=5
regexLineWithProduct = "^(?<prodlabel>.*?)(?:\\s+)(?<prodcode>Code-\\w+)\$" // eg Code-00260B
regexLineWithReportDate = "^(?:Disaggregated.*?),\\s(?<reportdate>.*?)\\s+\\n-+\$" // NB: contains \n
regexLineWithPositionsAll = "^(?:[\\s:]+Positions[\\s:]+\\n)All.*?:\\s*(?:[\\s\\d,]*?):(?:\\s*[\\d,]*){$skipFields}\\s*(?<managedLong>[\\d,]*)\\s*(?<managedShort>[\\d,]*).*\$" // NB: contains \n
regexLineWithPositionsChanges = "^(?:[\\s:]+Changes in Commitments from[^\\n]+\\n).*?:\\s*(?:[\\s\\d,\\-]*?):(?:\\s*[\\d,\\-]*){$skipFields}\\s*(?<managedLongChg>[\\d,\\-]*)\\s*(?<managedShortChg>[\\d,\\-]*).*\$" // NB: contains \n
regex = [regexLineWithProduct,regexLineWithReportDate,regexLineWithPositionsAll,regexLineWithPositionsChanges].join("|")
int row=0
Closure stagger = { row += (it.grp== "prodlabel") ? 1:0; return [(it.grp):it.s, "row":row ] }
Closure merge = { Integer sn, List singleKeyMaps -> singleKeyMaps.inject(["row": sn]) { Map accum, Map curr -> accum << curr } }
List<Map> tokens = get_matches(fileContents,regex,"prodlabel,prodcode,reportdate,OI,managedShort")
List<Map> listWithLotNumber = tokens.collect stagger
Map<Integer,List<Map>> lots = listWithLotNumber.groupBy { it.row }
List<Map> out = lots.collect merge
println ""
delim="|" // assuming no escaping is required
println "row,prodlabel,prodcode,reportdate,managedLong,managedShort,managedLongChg,managedShortChg,PrevBullBearRatio,BullBearRatio,BullBearRatioChange".replaceAll(",",delim)
// enrich with signals:
out = out.collect { Map map ->
managedShort = map.managedShort ? Float.parseFloat(map.managedShort.replaceAll(",", "")) : 0 // not required
managedLong = map.managedLong ? Float.parseFloat(map.managedLong.replaceAll(",", "")) : 0 // not required
managedLongChg = map.managedLongChg ? Float.parseFloat(map.managedLongChg.replaceAll(",", "")) : 0 // may be dot in CORN
managedShortChg = map.managedShortChg ? Float.parseFloat(map.managedShortChg.replaceAll(",", "")) : 0 // may be dot in CORN
BullBearRatio = (managedLong+managedShort==0) ? 0 : managedLong / (managedLong+managedShort)
// [Number of Bullish views/ (Bullish + Bearish views)] // x 100
// computing prev-week values: if chg reads -3, prev-week value = this+3 = this - change
managedLong = managedLong - managedLongChg
managedShort = managedShort - managedShortChg
PrevBullBearRatio = (managedLong+managedShort==0) ? 0 : managedLong / (managedLong+managedShort)
BullBearRatioChange = "="
if(BullBearRatio.round(2) > PrevBullBearRatio.round(2)) BullBearRatioChange="+"
if(BullBearRatio.round(2) < PrevBullBearRatio.round(2)) BullBearRatioChange="-"
return map + [BullBearRatio:BullBearRatio.round(2), PrevBullBearRatio:PrevBullBearRatio.round(2), BullBearRatioChange:BullBearRatioChange ]
}
out.each { Map map -> println "row,prodlabel,prodcode,reportdate,managedLong,managedShort,managedLongChg,managedShortChg,PrevBullBearRatio,BullBearRatio,BullBearRatioChange".tokenize(",").collect { map[it] }.join(delim) }
The COT report with the tokens identified and highlighted by the regex defined in the Lexer above is available at this link.
A few tokens (like "Other:") are inaccurate but the extraction of the data that is relevant to our objective is not affected.
¶ output
below is the output of the groovy program, which contains both the regex logic and the computation of the Bull/Bear ratio and its one-week change.
row|prodlabel|prodcode|reportdate|managedLong|managedShort|managedLongChg|managedShortChg|PrevBullBearRatio|BullBearRatio|BullBearRatioChange
1|WHEAT-SRW - CHICAGO BOARD OF TRADE|Code-001602|May 02, 2023|58,776|185,100|551|13,862|0.25|0.24|-
2|WHEAT-HRW - CHICAGO BOARD OF TRADE|Code-001612|May 02, 2023|38,707|44,171|938|13,773|0.55|0.47|-
3|WHEAT-HRSpring - MINNEAPOLIS GRAIN EXCHANGE|Code-001626|May 02, 2023|4,844|13,050|-2,430|2,366|0.41|0.27|-
4|CORN - CHICAGO BOARD OF TRADE|Code-002602|May 02, 2023|152,174|270,320|-38,506|64,343|0.48|0.36|-
5|CORN CONSECUTIVE CSO - CHICAGO BOARD OF TRADE|Code-00260B|May 02, 2023|0|7|null|null|0.0|0.0|=
6|OATS - CHICAGO BOARD OF TRADE|Code-004603|May 02, 2023|336|1,738|0|312|0.19|0.16|-
7|ROUGH RICE - CHICAGO BOARD OF TRADE|Code-039601|May 02, 2023|603|1,309|118|-186|0.24|0.32|+
8|LEAN HOGS - CHICAGO MERCANTILE EXCHANGE|Code-054642|May 02, 2023|52,260|59,083|2,406|-11,433|0.41|0.47|+
9|LIVE CATTLE - CHICAGO MERCANTILE EXCHANGE|Code-057642|May 02, 2023|124,263|16,712|-1,597|-2,115|0.87|0.88|+
10|FEEDER CATTLE - CHICAGO MERCANTILE EXCHANGE|Code-061641|May 02, 2023|21,835|7,248|918|-460|0.73|0.75|+
11|BUTTER (CASH SETTLED) - CHICAGO MERCANTILE EXCHANGE|Code-050642|May 02, 2023|55|176|55|-2|0.0|0.24|+
12|MILK, Class III - CHICAGO MERCANTILE EXCHANGE|Code-052641|May 02, 2023|0|5,293|0|1,727|0.0|0.0|=
13|NON FAT DRY MILK - CHICAGO MERCANTILE EXCHANGE|Code-052642|May 02, 2023|90|306|0|-21|0.22|0.23|+
14|CME MILK IV - CHICAGO MERCANTILE EXCHANGE|Code-052644|May 02, 2023|318|0|3|0|1.0|1.0|=
15|CHEESE (CASH-SETTLED) - CHICAGO MERCANTILE EXCHANGE|Code-063642|May 02, 2023|0|2,086|0|425|0.0|0.0|=
16|SOYBEANS - CHICAGO BOARD OF TRADE|Code-005602|May 02, 2023|107,640|51,267|-7,687|23,148|0.8|0.68|-
17|SOYBEAN OIL - CHICAGO BOARD OF TRADE|Code-007601|May 02, 2023|53,844|77,578|4,535|8,714|0.42|0.41|-
18|SOYBEAN MEAL - CHICAGO BOARD OF TRADE|Code-026603|May 02, 2023|83,479|22,922|-24,047|1,769|0.84|0.78|-
19|USD Malaysian Crude Palm Oil C - CHICAGO MERCANTILE EXCHANGE|Code-037021|May 02, 2023|0|240|0|40|0.0|0.0|=
20|CANOLA - ICE FUTURES U.S.|Code-135731|May 02, 2023|18,867|81,022|-1,746|8,708|0.22|0.19|-
21|COTTON NO. 2 - ICE FUTURES U.S.|Code-033661|May 02, 2023|28,913|50,801|-546|1,043|0.37|0.36|-
22|FRZN CONCENTRATED ORANGE JUICE - ICE FUTURES U.S.|Code-040701|May 02, 2023|4,430|476|31|-19|0.9|0.9|=
23|COCOA - ICE FUTURES U.S.|Code-073732|May 02, 2023|111,952|62,461|-3,786|206|0.65|0.64|-
24|SUGAR NO. 11 - ICE FUTURES U.S.|Code-080732|May 02, 2023|260,090|52,290|-18,498|2,381|0.85|0.83|-
25|COFFEE C - ICE FUTURES U.S.|Code-083731|May 02, 2023|45,710|13,704|-1,663|1,411|0.79|0.77|-
¶ useful links about regex:
https://www.rexegg.com/regex-quickstart.html
https://www.regular-expressions.info/refquick.html
https://www.regular-expressions.info/refcapture.html
https://www.regular-expressions.info/refadv.html
http://www.rexegg.com/regex-lookarounds.html
(?=foo) (Positive) Lookahead:
Asserts that what immediately follows the current position in the string is foo
(?<=foo) (Positive) Lookbehind:
Asserts that what immediately precedes the current position in the string is foo
(?!foo) Negative Lookahead:
Asserts that what immediately follows the current position in the string is not foo
(?<!foo) Negative Lookbehind:
Asserts that what immediately precedes the current position in the string is not foo
¶ Generic tokenizer + sequential pattern mining (ML)
We now look into an alternative approach in which, instead of leveraging on regex development skills, we will use a generic tokenizer that will generate a non-regular stream of tokens that we will be processed by a sequential pattern mining algorithm: we will manually tag a few tokens to generate a training set and we will run the CRF++ ML algorithm (conditional random field) on the test set to have all the tokens we need extracted from the stream.
Links:
http://www.philippe-fournier-viger.com/spmf/index.php?link=algorithms.php
https://taku910.github.io/crfpp/
Generic tokenizer (non regex-based): JavaScanner
// this is streamtknz.groovy
assert args.size()==1
String infilename= args[0]
class Xscanner {
Scanner sc
int inline_token_sn
int lineno
Xscanner(String line, int lineno_) {
inline_token_sn=0
lineno=lineno_
sc= new Scanner(line) //https://docs.oracle.com/javase/1.5.0/docs/api/java/util/Scanner.html
}
Xscanner() {}
String safeprint(String scnext) {
int stkzttype = (int)scnext[0]
return stkzttype>=32 && stkzttype<255 ? sprintf("%c",stkzttype) : "N/P"
}
void xprint(List L) {
if(L==null) { println(["lineno","LN_TK_SN","tkcode","tkval"].join("\t")); return } // print headers
assert L.size()==2
println( ([lineno, inline_token_sn]+L).join("\t"))
}
void consume() {
if(sc.hasNextDouble()) {
xprint(["TT_NUM", sc.nextDouble()])
} else {
String scnext = sc.next()
if(scnext.size()>1) {
xprint(["TT_WORD", scnext])
} else {
int stkzttype = (int)scnext[0]
xprint(["ASCII"+stkzttype, safeprint(scnext)])
}
}
inline_token_sn++
}
}
new Xscanner().xprint()
int lineno
new File(infilename).eachLine("UTF-8") { line ->
Xscanner p = new Xscanner(line,lineno++)
while(p.sc.hasNext())
p.consume()
p.xprint(["TT_EOL", "N/P"])
}
execution:
groovy streamtknz.groovy CFTC_COT_report.txt > CFTC_COT_stream_of_tokens.tsv
output with training labels added (excerpt):
| inputLineSeqNo | LineTokenSeqNo | TokenCode | TokenText | TrainingLabel |
|---|---|---|---|---|
| 0 | 0 | TT_EOL | N/P | TT_EOL |
| 1 | 0 | TT_WORD | WHEAT-SRW | PRODUCT_NAME |
| 1 | 1 | ASCII45 | - | ASCII45 |
| 1 | 2 | TT_WORD | CHICAGO | TT_WORD |
| 1 | 3 | TT_WORD | BOARD | TT_WORD |
| 1 | 4 | TT_WORD | OF | TT_WORD |
| 1 | 5 | TT_WORD | TRADE | TT_WORD |
| 1 | 6 | TT_WORD | Code-001602 | PRODUCT_CODE |
| 1 | 7 | TT_EOL | N/P | TT_EOL |
| ... | ||||
| 4 | 1 | ASCII58 | : | ASCII58 |
| 4 | 2 | TT_WORD | Reportable | /Reportable |
| ... | ||||
| 6 | 7 | ASCII58 | : | ASCII58 |
| 6 | 8 | TT_WORD | Managed | /Managed |
| 6 | 9 | TT_WORD | Money | TT_WORD |
| 6 | 10 | ASCII58 | : | ASCII58 |
| ... | ||||
| 7 | 11 | ASCII58 | : | ASCII58 |
| 7 | 12 | TT_WORD | Long | LONG_HEADER |
| 7 | 13 | ASCII58 | : | ASCII58 |
| 7 | 14 | TT_WORD | Short | SHORT_HEADER |
| 7 | 15 | TT_WORD | :Spreading | TT_WORD |
| ... | ||||
| 11 | 7 | TT_NUM | 14141 | TT_NUM |
| 11 | 8 | TT_NUM | 58776 | MMLONGVAL |
| 11 | 9 | TT_NUM | 185100 | MMSHORTVAL |
| 11 | 10 | TT_NUM | 70898 | TT_NUM |
| ... | ||||
| 35 | 8 | TT_NUM | 54.5 | TT_NUM |
| 35 | 9 | TT_EOL | N/P | TT_EOL |
| 36 | 0 | TT_EOL | N/P | TT_EOL |
| ... | ||||
| 73 | 0 | TT_WORD | WHEAT-HRSpring | PRODUCT_NAME |
| 73 | 1 | ASCII45 | - | ASCII45 |
| 73 | 2 | TT_WORD | MINNEAPOLIS | TT_WORD |
| 73 | 3 | TT_WORD | GRAIN | TT_WORD |
| 73 | 4 | TT_WORD | EXCHANGE | TT_WORD |
| 73 | 5 | TT_WORD | Code-001626 | ??? (expected prediction label: PRODUCT_CODE) |
| 73 | 6 | TT_EOL | N/P | TT_EOL |
NB
the Exchange of the first product was tokenized into 4 words: CHICAGO-BOARD-OF-TRADE;
the Exchange of the second product was tokenized into 3 words: MINNEAPOLIS-GRAIN-EXCHANGE;
in broader terms, the number of tokens generated for each commodity product may vary
between 387 and 394;
hence the need of an ML algorithm trained to cope with such irregular sequences
We keep our conditional random field definitions as naive as possible, based on the token type and its horizontal position (i.e. how many preceding tokens occur in the same line) and vertical offset (w.r.t. the beginning of the current "commodity product table"), as follows:
# CRF_TEMPLATE.txt
# Unigram
U_tokenclass_line_pos__prev2:%x[-2,1]/%x[-2,2]/%x[-2,3]
U_tokenclass_line_pos__prev1:%x[-1,1]/%x[-1,2]/%x[-1,3]
U_tokenclass_line_pos__curr_:%x[0,1]/%x[0,2]/%x[0,3]
U_tokenclass_line_pos__next1:%x[1,1]/%x[1,2]/%x[1,3]
U_tokenclass_line_pos__next2:%x[2,1]/%x[2,2]/%x[2,3]
We run this workflow:
# Makefile
# reminder: $@ = the target
all: CFTC_COT_stream_of_tokens.predictions-model1.extract.tsv CFTC_COT_stream_of_tokens.predictions-model2.extract.tsv
CFTC_COT_stream_of_tokens.tsv : streamtknz.groovy CFTC_COT_report.txt
groovy streamtknz.groovy CFTC_COT_report.txt > $@
CRF_TEMPLATE = CRF_TEMPLATE.txt
model1 : CFTC_COT_stream_of_tokens.trainset.tsv $(THE_TEMPLATE)
@echo running LEARNER 1 :
.\CRF\CRFpp-058\crf_learn -f 3 -c 4.0 $(CRF_TEMPLATE) CFTC_COT_stream_of_tokens.trainset.tsv $@
model2 : CFTC_COT_stream_of_tokens.trainset.tsv $(THE_TEMPLATE)
@echo running LEARNER 2 :
.\CRF\CRFpp-058\crf_learn -a MIRA -f 3 $(CRF_TEMPLATE) CFTC_COT_stream_of_tokens.trainset.tsv $@
CFTC_COT_stream_of_tokens.predictions-model1.tsv : CFTC_COT_stream_of_tokens.testset.tsv model1
@echo running the PREDICTOR 1 :
.\CRF\CRFpp-058\crf_test -m model1 CFTC_COT_stream_of_tokens.testset.tsv model1 > $@
CFTC_COT_stream_of_tokens.predictions-model2.tsv : CFTC_COT_stream_of_tokens.testset.tsv model2
@echo running the PREDICTOR 2 :
.\CRF\CRFpp-058\crf_test -m model2 CFTC_COT_stream_of_tokens.testset.tsv model2 > $@
CFTC_COT_stream_of_tokens.predictions-model1.extract.tsv : parse_prediction.awk CFTC_COT_stream_of_tokens.predictions-model1.tsv
@echo extracting the key PREDICTIONS - model-1:
awk -f parse_prediction.awk CFTC_COT_stream_of_tokens.predictions-model1.tsv > $@
CFTC_COT_stream_of_tokens.predictions-model2.extract.tsv : parse_prediction.awk CFTC_COT_stream_of_tokens.predictions-model2.tsv
@echo extracting the key PREDICTIONS - model-2:
awk -f parse_prediction.awk CFTC_COT_stream_of_tokens.predictions-model2.tsv > $@
the final raw output, where we can see the Long and Short Open Interests held by the Managed Money for each commodity, is as follows:
1 PRODUCT_NAME WHEAT-SRW PRODUCT_CODE Code-001602 MMLONG 58776.0 MMSHORT 185100.0
2 PRODUCT_NAME WHEAT-HRW PRODUCT_CODE Code-001612 MMLONG 38707.0 MMSHORT 44171.0
3 PRODUCT_NAME WHEAT-HRSpring PRODUCT_CODE Code-001626 MMLONG 4844.0 MMSHORT 13050.0
4 PRODUCT_NAME CORN PRODUCT_CODE Code-002602 MMLONG 152174.0 MMSHORT 270320.0
5 PRODUCT_NAME CORN PRODUCT_CODE Code-00260B MMLONG 0.0 MMSHORT 7.0
6 PRODUCT_NAME OATS PRODUCT_CODE Code-004603 MMLONG 336.0 MMSHORT 1738.0
7 PRODUCT_NAME ROUGH PRODUCT_CODE Code-039601 MMLONG 603.0 MMSHORT 1309.0
8 PRODUCT_NAME LEAN PRODUCT_CODE Code-054642 MMLONG 52260.0 MMSHORT 59083.0
9 PRODUCT_NAME LIVE PRODUCT_CODE Code-057642 MMLONG 124263.0 MMSHORT 16712.0
10 PRODUCT_NAME FEEDER PRODUCT_CODE Code-061641 MMLONG 21835.0 MMSHORT 7248.0
11 PRODUCT_NAME BUTTER PRODUCT_CODE Code-050642 MMLONG 55.0 MMSHORT 176.0
12 PRODUCT_NAME MILK, PRODUCT_CODE Code-052641 MMLONG 0.0 MMSHORT 5293.0
13 PRODUCT_NAME NON PRODUCT_CODE Code-052642 MMLONG 90.0 MMSHORT 306.0
14 PRODUCT_NAME CME PRODUCT_CODE Code-052644 MMLONG 318.0 MMSHORT 0.0
15 PRODUCT_NAME CHEESE PRODUCT_CODE Code-063642 MMLONG 0.0 MMSHORT 2086.0
16 PRODUCT_NAME SOYBEANS PRODUCT_CODE Code-005602 MMLONG 107640.0 MMSHORT 51267.0
17 PRODUCT_NAME SOYBEAN PRODUCT_CODE Code-007601 MMLONG 53844.0 MMSHORT 77578.0
18 PRODUCT_NAME SOYBEAN PRODUCT_CODE Code-026603 MMLONG 83479.0 MMSHORT 22922.0
19 PRODUCT_NAME USD PRODUCT_CODE Code-037021 MMLONG 0.0 MMSHORT 240.0
20 PRODUCT_NAME CANOLA PRODUCT_CODE Code-135731 MMLONG 18867.0 MMSHORT 81022.0
21 PRODUCT_NAME COTTON PRODUCT_CODE Code-033661 MMLONG 28913.0 MMSHORT 50801.0
22 PRODUCT_NAME FRZN PRODUCT_CODE Code-040701 MMLONG 4430.0 MMSHORT 476.0
23 PRODUCT_NAME COCOA PRODUCT_CODE Code-073732 MMLONG 111952.0 MMSHORT 62461.0
24 PRODUCT_NAME SUGAR PRODUCT_CODE Code-080732 MMLONG 260090.0 MMSHORT 52290.0
25 PRODUCT_NAME COFFEE PRODUCT_CODE Code-083731 MMLONG 45710.0 MMSHORT 13704.0
26 PRODUCT_NAME ------------------------------------------------------------------------ PRODUCT_CODE MMLONG MMSHORT
the final output contains:
25 true positives,
1 false positive,
0 true negatives,
0 false negatives.
¶
back to Portfolio
