There are a variety of LLM leaderboards for reference. On the open side, this include HF's "Open LLM Leaderboard" ([https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard)](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard)) and LMSys's "Chatbot Arena LLM Leaderboard" ([https://lmarena.ai/?leaderboard).](https://lmarena.ai/?leaderboard).) On the closed side, Scale's "Leaderboard" is relatvely new and seems to be gaining traction ([https://scale.com/leaderboard).](https://scale.com/leaderboard).) You can filter by different benchmarks, but they all have various math components.

If you're choosing between frontier models, I'd just use the platform with the best UX for your use case. In general, frontier models all perform well in math (and show little sign of overfitting on the evaulation benchmarks - [https://arxiv.org/html/2405.00332v3),](https://arxiv.org/html/2405.00332v3),) including OpenAI's o1-preview, Claude's 3.5 Sonnet, Gemini's 1.5 Pro, and Llama's 3.1 405B Instruct. 

Both OpenAI ([https://openai.com/index/improvements-to-data-analysis-in-chatgpt/)](https://openai.com/index/improvements-to-data-analysis-in-chatgpt/)) and Anthropic (more recently, [https://www.anthropic.com/news/analysis-tool)](https://www.anthropic.com/news/analysis-tool)) have got data analysis tools built-in, and Hugging Face ([https://huggingface.co/chat/)](https://huggingface.co/chat/)) has the same for open models.

There are also open source tools where you can compare different frontier models at once, which gives you a sense check for the variety of responses - [https://github.com/nat/openplayground?tab=readme-ov-file.](https://github.com/nat/openplayground?tab=readme-ov-file.)

If you're choosing between OpenAI models, then o1-preview is leading with it's advanced reasoning. You can compare different OpenAI models and parameter configurations here - [https://platform.openai.com/playground/chat?models=gpt-4o-mini-2024-07-18.](https://platform.openai.com/playground/chat?models=gpt-4o-mini-2024-07-18.)

References:

- [https://huggingface.co/spaces/open-llm-leaderboard/blog](https://huggingface.co/spaces/open-llm-leaderboard/blog)
- [https://lmarena.ai/?leaderboard](https://lmarena.ai/?leaderboard)
  - [https://blog.lmarena.ai/blog/2023/arena/](https://blog.lmarena.ai/blog/2023/arena/)
  - [https://github.com/lm-sys/FastChat](https://github.com/lm-sys/FastChat)
- [https://scale.com/leaderboard](https://scale.com/leaderboard)
  - [https://x.com/alexandr_wang/status/1795857651592491281](https://x.com/alexandr_wang/status/1795857651592491281)
  - [https://x.com/karpathy/status/1795873666481402010](https://x.com/karpathy/status/1795873666481402010)
  - [https://arxiv.org/html/2405.00332v3](https://arxiv.org/html/2405.00332v3)
- MATH dataset / benchmark
  - [https://arxiv.org/abs/2103.03874](https://arxiv.org/abs/2103.03874)

LLM leaderboard and evaluations

Drives:

- local drives
  - storage devices (HDD / SSD / USB) physically connected to a computer
- network drives
  - storage devices not physcially connected to a computer but accessed over a network
  - usually part of a network-attached storage (NAS) system or a dedicated file server
- shared drives
  - local drives or network drives that are shared across a network to allow multiple users to access them

Clients for SSH / SFTP / FTP:

- WinSCP (file transfer)
  - [https://winscp.net/eng/download.php](https://winscp.net/eng/download.php)
  - [https://github.com/winscp/winscp](https://github.com/winscp/winscp)
- Putty (ssh)
  - [https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html](https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html)

Mapping network drives (on Linux):

- NFS
  - [https://ubuntu.com/server/docs/network-file-system-nfs](https://ubuntu.com/server/docs/network-file-system-nfs)
- FUSE
  - [https://github.com/libfuse/libfuse](https://github.com/libfuse/libfuse) (FUSE)
    - an interface for userspace programs to export a filesystem to the Linux kernel
- SSHFS
  - [https://github.com/libfuse/sshfs](https://github.com/libfuse/sshfs)
  - network filesystem client to connect to SSH servers

Mapping network drives (on Windows):

- Samba
  - [https://www.samba.org/](https://www.samba.org/)
  - [https://github.com/samba-team/samba](https://github.com/samba-team/samba)
- SSHFS-WIN (Windows File System Proxy)
  - [https://github.com/winfsp/sshfs-win](https://github.com/winfsp/sshfs-win)
    - [https://cygwin.com/](https://cygwin.com/) (for POSIX environment)
    - [https://github.com/winfsp/winfsp](https://github.com/winfsp/winfsp) (for FUSE)
  - the README example maps a network drive from a Macbook

Tools on Windows:

- check which shared drives exist:
  - `Computer Management` > `System Tools` > `Shared Folders` > `Shares`

Local drives, network drives, and ssh / file clients for servers

DER (`Distinguished Encoding Rules`) files:

- results in a truly binary representation of the encoded data

PEM (`Privacy Enhanced Mai`) files:

- essentially base64 encoded versions of the DER encoded data (a format to send these in an encoding of printable characters)
- usually generated by `OpenSSL` and composed of a header, the base64 encoded data, and a footer
- format
  - a line consisting of 5 hyphens, the word BEGIN, one or a few (space-separated) words defining the type of data, and 5 hyphens
  - an optional (and rare) rfc822-style header, terminated by an empty line
  - base64 of the data, broken into lines of 64 characters (except the last); some programs instead use the (slightly newer) MIME limit of 76 characters
  - a line like the BEGIN line but with END instead

RSA Private Key file (`PKCS#1`):

- starts with `-----BEGIN RSA PRIVATE KEY-----`
- essentially just the key object from PKCS#8, but without the version or algorithm identifier in front
- contains CRT parameters

```
RSAPrivateKey ::= SEQUENCE {
  version           Version,
  modulus           INTEGER,  -- n
  publicExponent    INTEGER,  -- e
  privateExponent   INTEGER,  -- d
    "prime1NTEGER,  --",
    "prime2NTEGER,  --",
  exponent1         INTEGER,  -- d mod (p-1)
  exponent2         INTEGER,  -- d mod (q-1)
  coefficient       INTEGER,  -- (inverse of q) mod p
  otherPrimeInfos   OtherPrimeInfos OPTIONAL
}
```

Private Key file (`PKCS#8`):

- starts with `-----BEGIN PRIVATE KEY -----`
- this is a more generic key format that identifies the type of public key and contains the relevant data
  - useful because `RSA` is not used exclusively inside `X509` and `SSL/TLS`
- does not contain CRT parameters (more headers, but smaller size)
- `OpenSSL` version 3.0.0 and later use this format by default

```crt
PrivateKeyInfo ::= SEQUENCE {
  version         Version,
  algorithm       AlgorithmIdentifier,
  PrivateKey      BIT STRING
}

AlgorithmIdentifier ::= SEQUENCE {
  algorithm       OBJECT IDENTIFIER,
  parameters      ANY DEFINED BY algorithm OPTIONAL
}
```

Practical observations:

- for public keys
  - open keystore file
  - we use the head certificate from the PFX file when deploying on servers (there are three certificates in the certificate chain)
  - export as PEM encoding
- for private keys
  - we use `PKCS#8` encoding, which is marked by `-----BEGIN PRIVATE KEY-----`
  - export private key and certificates, PEM encoded

References:

- [https://github.com/scop/portecle](https://github.com/scop/portecle)
- [https://stackoverflow.com/questions/20065304/differences-between-begin-rsa-private-key-and-begin-private-key](https://stackoverflow.com/questions/20065304/differences-between-begin-rsa-private-key-and-begin-private-key)
- [https://web.archive.org/web/20140819203300/https://polarssl.org/kb/cryptography/asn1-key-structures-in-der-and-pem](https://web.archive.org/web/20140819203300/https://polarssl.org/kb/cryptography/asn1-key-structures-in-der-and-pem)
- [https://stackoverflow.com/questions/48958304/pkcs1-and-pkcs8-format-for-rsa-private-key](https://stackoverflow.com/questions/48958304/pkcs1-and-pkcs8-format-for-rsa-private-key)

Portecle, DER and PEM files, and PEM encodings for private key files

- [https://docs.pytest.org/en/latest/how-to/fixtures.html#teardown-cleanup-aka-fixture-finalization](https://docs.pytest.org/en/latest/how-to/fixtures.html#teardown-cleanup-aka-fixture-finalization)

```python

@pytest.fixture(scope="session", autouse=True)
def fixt_datastore(
    fixt_datastore_settings: DatabaseConnectionSettings,
) -> Generator[Datastore, None, None]:
    datastore = create_datastore(fixt_datastore_settings)

    with datastore._session_factory.get_session() as session:
        engine = session.get_bind()
        assert engine.url.drivername == "sqlite"

    datastore.create_all_models()
    yield datastore
    datastore.drop_all_models()
```

Autouse:

- scope mismatch (especially with tmp_path fixture) can cause issues, use fixture factories to avoid this
  - `ScopeMismatch: You tried to access the function scoped fixture tmp_path with a session scoped request object, involved factories:`
  - [https://github.com/tortoise/tortoise-orm/issues/638](https://github.com/tortoise/tortoise-orm/issues/638)
  - [https://docs.pytest.org/en/6.2.x/fixture.html#autouse-order](https://docs.pytest.org/en/6.2.x/fixture.html#autouse-order)

Fixture factory:


Sharing in-memory sqlite database across tests:

- [https://stackoverflow.com/questions/32681761/how-can-i-attach-an-in-memory-sqlite-database-in-python/32681822#32681822](https://stackoverflow.com/questions/32681761/how-can-i-attach-an-in-memory-sqlite-database-in-python/32681822#32681822)
- when you use datastore as a fixture, it will be created and destroyed for each test that uses it, the tables won't be present across tests, but we can attach an in-memory database to the sqlite engine to share the tables across tests

Fixture set up and tear down in pytest

- overviews
  - pandas 2.0 moved from numpy backend (c++) to pyarrow backend
  - polars is written in rust with python bindings
  - both implement the same Apache Arrow data specification (Pandas 2.0 uses pyarrow, Polars uses arrow2)
- data format and performance
  - in-memory, contiguous columnar format with a rich data type system (including nested and user-defined data types)
  	- supports vectorization using the latest SIMD (Single Instruction, Multiple Data) operations in modern processors
  	- supports zero-copy reads for lightning-fast data access without serialization overhead
  	- supports computational routines and execution engines to maximize their efficiency
- standardisation
  - without a standard columnar data format, every technology (languages and databases) has to implement its own
  	- moving data involves costly serialization and deserialization
  - with arrow, you can share data in an in-memory representation without persisting
  	- don't need to create custom connectors
  	- facilitates reuse of libraries of algorithms, across languages
- ecosystem
  - a language-agnostic framework with a vast ecosystem
  	- official libraries across programming languages implement the format
  		- third-party projects may work with arrow data without having to implement the format themselves
  		- easy to share the data among different programs

Luke Miloszewski

Featured Posts

Featured Projects