Understanding Data Storage Formats

Data Storage Formats in JSPlots.jl

Specifying datasets

For most charttypes you have some dataset that they should comsume. As part of the constructor you give a symbol that identifies the dataset to be used. Then at the point when you construct a page you input a dict mapping these symbols to an object holding the data.

These symbols can have a period in it in which case you are describing a subfield. For instance you might have an object like that below:

struct ExampleStruct
    aa::DataFrame
    bb::DataFrame
end
example_struct = ExampleStruct(df1, df2)

Then if you have a chart that wants to use the aa dataset from this object then you can specify it with Symbol("example_struct.aa") and then later on you give the data in the dict as Dict{Symbol,Any}(:example_struct => example_struct).

The point of this system is twofold. For one certain charts may require datasets and so it is easier in this case to have a struct to hold everything together (Currently this is only ExecutionChart but maybe in the future). So if you wanted to put in a pivottable in the same page to show only the fills data you could do that without putting in the data twice. The second reason is that using structs in this way will bunch the data in different subfolders in cases where an external data format is used (:parquet, :json_external or :csv_external as described below).

Data Format

JSPlots.jl supports multiple data storage formats for embedding data in HTML pages. The format affects file size, loading speed, and browser compatibility.

Available Formats

1. :parquet (Recommended)

Best for: Large datasets, multi-page reports, production use

• Compressed binary format - smallest file sizes (typically 5-10x smaller than CSV)
• Fast loading - efficient decompression in browser
• Modern browsers - requires JavaScript support
• Column-oriented - excellent compression for tabular data

page = JSPlotPage(data_dict, charts, dataformat = :parquet)

2. :csv_embedded

Best for: Small datasets, maximum compatibility, debugging

• Text-based CSV format - human-readable in HTML source
• Single file - data embedded directly in HTML
• Compact format - smaller than JSON for tabular data
• Universal compatibility - works everywhere
• Easy debugging - can inspect data in HTML source
• No deduplication - data embedded separately in each page

page = JSPlotPage(data_dict, charts, dataformat = :csv_embedded)

3. :csv_external

Best for: Sharing data files, version control, spreadsheet analysis

• Separate CSV files - one file per dataset in data/ subdirectory
• Multiple files - HTML + CSV files
• Enables deduplication - shared datasets only stored once
• Data reusability - CSV files can be opened in Excel/spreadsheets
• Better for git - separate data from HTML

page = JSPlotPage(data_dict, charts, dataformat = :csv_external)

4. :json_embedded

Best for: Small datasets, web developers, API-like structure

• JSON format - familiar to web developers
• Single file - data embedded directly in HTML
• Structured data - preserves data types (numbers, strings, booleans)
• Larger than CSV - JSON is ~25% larger due to key names in each row
• Fast browser parsing - native JSON.parse() is highly optimized
• No deduplication - data embedded separately in each page

page = JSPlotPage(data_dict, charts, dataformat = :json_embedded)

5. :json_external

Best for: API consumption, web applications, data interchange

• Separate JSON files - one file per dataset in data/ subdirectory
• Multiple files - HTML + JSON files
• Enables deduplication - shared datasets only stored once
• API-ready format - can be consumed by other web applications
• Type preservation - maintains data types better than CSV
• Web-friendly - easy to load from external servers

page = JSPlotPage(data_dict, charts, dataformat = :json_external)

Data Deduplication

Key Efficiency Feature: JSPlots.jl automatically detects when the same dataset is used multiple times and only includes it once.

Note that thisData deduplication only works with external formats Data deduplication only works with external formats - (:parquet, :csv_external, :json_external). With embedded formats (:csv_embedded, :json_embedded), data is embedded separately in each HTML page, so there's no deduplication benefit.

How It Works

When you create a multi-page report with Pages, datasets are identified by their Symbol key in the data dictionary. The Pages-level dataformat setting overrides any individual JSPlotPage dataformat settings.

# Both pages use :sales_data
page1 = JSPlotPage(
    Dict(:sales_data => df),
    [chart1, chart2],
    dataformat = :csv_embedded  # This is ignored!
)

page2 = JSPlotPage(
    Dict(:sales_data => df),  # Same Symbol = reuses data!
    [chart3, chart4],
    dataformat = :json_embedded  # This is also ignored!
)

# Pages-level dataformat is what actually gets used
report = Pages(
    [intro],
    [page1, page2],
    dataformat = :parquet  # ← This overrides page-level settings!
)
# Result: One parquet file in data/sales_data.parquet shared by both pages

Benefits of Deduplication (External Formats Only)

• Reduced File Size: Large datasets only stored once, not duplicated per page
• Faster Loading: Less data to download and parse
• Memory Efficiency: Browser loads dataset into memory once
• Consistency: All pages use the exact same data

Format Comparison Table

Feature	:parquet	:csv_embedded	:csv_external	:json_embedded	:json_external
File Size	Smallest (5-10x smaller)	Medium	Medium	Large (~25% bigger than CSV)	Large (~25% bigger than CSV)
Loading Speed	Fast	Moderate	Moderate	Fast (native JSON.parse)	Fast (native JSON.parse)
Compatibility	Modern browsers	Universal	Universal	Universal	Universal
Deduplication	Yes (in Pages)	No (embedded)	Yes (in Pages)	No (embedded)	Yes (in Pages)
Type Preservation	Excellent	Limited (strings)	Limited (strings)	Good	Good
Human Readable	No (binary)	Yes (in HTML)	Yes (separate files)	Yes (in HTML)	Yes (separate files)
File Structure	Single HTML + data/	Single HTML	HTML + data/ folder	Single HTML	HTML + data/ folder
Best For	Production, large data	Small data, debugging	Spreadsheet analysis	Web developers	API/web apps

Setting Format Globally vs Per-Page

# Set format for entire report (RECOMMENDED)
report = Pages(
    [intro],
    [page1, page2],
    dataformat = :parquet  # Overrides any page-level settings
)

# Individual pages (only for single-page JSPlotPage, not Pages)
page1 = JSPlotPage(data_dict, charts, dataformat = :parquet)
create_html(page1, "standalone_page.html")

Note: When using Pages, the Pages-level dataformat overrides any dataformat settings in individual JSPlotPage objects.

Launcher Scripts for External Formats

When using external formats (:parquet, :csv_external, :json_external), JSPlots.jl automatically generates launcher scripts to help with browser security restrictions:

• open.sh - Shell script for Linux/Mac that launches a local web server
• open.bat - Batch script for Windows that opens with appropriate permissions
• README.md - Instructions for opening the HTML files

Why launcher scripts? Modern browsers block loading external files (data/, pictures/, etc.) from file:// URLs for security reasons. The launcher scripts either:

• Start a local web server (on Linux/Mac via Python's http.server)
• Launch the browser with flags to allow local file access (on Windows)

For embedded formats (:csv_embedded, :json_embedded), you can open the HTML file directly in a browser since all data is embedded - no launcher scripts needed!