This README is the one indexed from the repository at its latest ref, not from the tag for this version.

Medup

Sync all content from Medium or Dev.to with local folder via API

Table of Contents

• Usage
• Features
• Getting Started
• Contributing
• Contributors
• License

Usage

Export post

Export a single medium post to local

medup https://medium.com/notes-and-tips-in-full-stack-development/medup-backups-articles-8bf90179b094

Export a single dev.to post to local

medup https://dev.to/jetthoughts/the-trial-period-for-staff-augmentation-in-jetthoughts-1h5c

Author's posts

Export all articles written by single author to special folder posts/miry

medup -u miry -d posts/miry

Alternative:

medup @miry -d posts/miry

Similar for Dev.to

medup --platform=devto @miry
medup --platform=devto -u miry
medup https://dev.to/miry

User's recommendations

Export all articles recommended by miry:

medup -u miry -d posts/recommendations -r

Alternative:

medup @miry -d posts/recommendations -r

Publication's posts

Export all articles written by single author to special folder posts/miry

medup -p jetthoughts -d posts/jetthoughts

Alternative:

medup jetthoughts -d posts/jetthoughts

Similar for Dev.to

medup --platform=devto jetthoughts
medup --platform=devto -p jetthoughts
medup https://dev.to/jetthoughts

Assets

By default all images are embedded in the article. It is possible to save images in separate folder with arguments - --assets-images

$ medup @miry --assets-images -d posts/miry
$  $ tree posts/miry
posts/miry
├── 2017-03-20-war-with-ads-and-trackers.md
├── 2017-03-26-setup-k8s-v1-6-0-rc-1-cluster.md
├── 2017-03-28-docker-network-performance.md
...
└── assets
    ├── 0*BmauN6EER-PdvCee.jpeg
    ├── 0*FbFs8aNmqNLKw4BM.jpeg
    ├── 0*LZaURw4xtfA74nu9.jpeg
...

Specify assets directory

To make it is even more agile and work with Jekyl, put assets in different directory - --assets-dir=DIR

$ medup @miry --assets-images -d site/posts --assets-dir=site/assets
$ tree site
site
├── assets
│   ├── 0*BmauN6EER-PdvCee.jpeg
│   ├── 0*FbFs8aNmqNLKw4BM.jpeg
│   ├── 0*LZaURw4xtfA74nu9.jpeg
...
└── posts
    ├── 2017-03-20-war-with-ads-and-trackers.md
    ├── 2017-03-26-setup-k8s-v1-6-0-rc-1-cluster.md
    ├── 2017-03-28-docker-network-performance.md
...

Specify assets base path

Also change the path in the document to find assets with --assets-base-path=URI. Default base path is ./assets/ASSET_NAME.XXX. Example image: ![](./assets/1*hBBZVQZHFYJ0WWaj08bHuA.jpeg). With --assets-base-path=/static the src would be /static/ASSET_NAME.XXX. Example image: ![](/static/1*hBBZVQZHFYJ0WWaj08bHuA.jpeg).

$ medup @miry --assets-images -d tmp/posts --assets-dir=tmp/static --assets-base-path=/static

ENVIRONMENT

• MEDUP_GITHUB_API_TOKEN Use this GitHub personal access token when accessing the GitHub Gist.

Mix

The neat feature to mix mutliple rules in single command

medup -u miry @pftg -p jetthoughts notes-and-tips-in-full-stack-development \
      https://medium.com/notes-and-tips-in-full-stack-development/medup-backups-articles-8bf90179b094 \
      https://dev.to/jetthoughts \
      https://dev.to/jetthoughts/the-trial-period-for-staff-augmentation-in-jetthoughts-1h5c

It would download all articles for:

1. user @miry
2. user @pftg
3. publication jetthoughts
4. publication notes-and-tips-in-full-stack-development
5. single post "Medup backups articles"
6. publication jetthoughts from dev.to
7. single post "The Trial Period for Staff Augmentation in JetThoughts" from dev.to

Features

• Discover all articles from user account available in public
• Allow to download all recommended articles by user
• Discover all articles from publisher, that available in public
• Download images used inside article or in assets folder
• Save posts in markdown format
• Convert a single article in markdown format
• Extract gists and convert to code blocks

Getting Started

Homebrew

Install medup tool to the MacOS or Linux via Hombrew:

$ brew tap miry/medup
$ brew install medup

Run export command for Medium author miry and save articles to local folder:

$ medup -u miry -d .
$ medup -d . https://medium.com/notes-and-tips-in-full-stack-development/medup-backups-articles-8bf90179b094

Docker

Docker way to make same job:

$ docker run -v <path to local articles folder>:/posts -it miry/medup -u <user>

Download all articles that user has recommended:

$ docker run -v <path to local articles folder>:/posts -it miry/medup -u <user> --recommended -d posts/recommended

Crystal

Run dumping with source code

$ shards install
$ crystal run src/cli.cr -- -u <medium user> -d <destination folder>

Example download all posts from author https://medium.com/@miry to local folder posts/miry

$ crystal run src/cli.cr -- -u miry -d posts/miry

Build

Build a application binary and execute:

$ shards install
$ rake build
$ _output/medup -u <medium user> -d <destination folder>

Result

By default all articles destination folder is posts.

In the directory, you can find 2 format of files: .json and .md.

• JSON format is the raw, what Medium returns.
• Markdown format is simple implementation of block formated text.

Images encoded in the result document. Rich elements content are located in posts/assets and integrated with <IFRAME>.

Markdown

Example of exported document.

1. Original Medium post vs Exported Markdown:
2. Exported Markdown in Github pages vs original Dev.to

Demo

There are possible solution to test export markdown files. Run a test Jekyll server with posts. Example:

$ rake demo:serve

And after it finishes you can test http://localhost:4000. Update posts for demo:

$ medup -v7 -d demo/_posts/ --assets-dir=demo/assets --assets-base-path=/assets <url>|<@user>|<publisher>

Test Bridgetown use next command:

$ rake demo:bridgetown:serve

Bridgetown has a bit different structure for posts. Use to update posts:

$ medup -v7 -d demo/src/_posts/ --assets-dir=demo/src/assets --assets-base-path=/assets <url>|<@user>|<publisher>

Contributing

1. Fork it ( https://github.com/miry/medup/fork )
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create a new Pull Request

Contributors

• @miry Michael Nikitochkin - creator, maintainer
• @alexanderadam Alexander - supporter
• @clawfire Thibault Milan - supporter

License

This project is under the LGPL-3.0 license.

Play ownself

FAQ

• Clean unused assets:

$ for i in $(ls -1 assets/*); do grep -q "$i" *.md || echo $i; done | xargs rm

• Detect potential articles with IFRAME content for gists:

$ for i in $(ls -1 assets/*.html); do grep "gist.github.com" -q $i && grep -n "$i" *.md; done

• Detect potential articles' url with IFRAME content for gists:

$ for i in $(ls -1 assets/*.html); do grep "gist.github.com" -q $i && (grep -l "$i" *.md | xargs head | grep url); done

• Detect potential articles' url with IFRAME content for youtube:

$ for i in $(ls -1 assets/*.html); do grep "schema=youtube" -q $i && (grep -l "$i" *.md | xargs head | grep url); done

Medium API

1. Medium API documentation

2. Generate token on the page https://medium.com/me/settings

3. Create environment variable MEDIUM_TOKEN=<token>

4. Verify token with sample query:

$ curl -H "Authorization: Bearer <token>" https://api.medium.com/v1/me
{"data":{"id":"number","username":"miry","name":"Michael Nikitochkin","url":"https://medium.com/@miry","imageUrl":"https://cdn-images-1.medium.com/fit/c/400/400/0*KgbjgGnH-csHuB8j."}}

Crawler

1. Check public information not covered by API

$ curl "https://medium.com/@miry?format=json" | cut -c17- && : Remove in the front from response some strange JS code.

2. Pagination

$ curl "https://medium.com/@miry/latest?format=json&limit=100" | cut -c17- && : Remove in the front from response some strange JS code.

3. Post info

curl -s -H "Content-Type: application/json" https://medium.com/@miry/c35b40c499e\?format\=json\&limit\=100

4. Stream

$ curl -s -H "Content-Type: application/json" "https://medium.com/_/api/users/fdf238948af6/profile/stream?source=overview" | cut -c17-
$ curl -s -H "Content-Type: application/json" "https://medium.com/_/api/users/fdf238948af6/profile/stream?limit=100&page=3" | cut -c17- > stream.json
$ cat stream.json| jq ".payload.references.Post[].title"
$ cat stream.json| jq ".payload.paging.next"

5. Recommendations

$ curl -s -H "Content-Type: application/json" https://medium.com/@miry/has-recommended | cut -c17-
$ curl -s -H "Content-Type: application/json" "https://medium.com/_/api/users/fdf238948af6/profile/stream?source=has-recommended" | cut -c17-