feat: cloudflare support

CONTRIBUTING.md

@@ -6,3 +6,9 @@ Before doing any big contributions it is good idea to first discuss it in releva
 direction is the best and what sort of result will be accepted best.
 
 Also it is welcomed if your PRs will contain test coverage.
+
+## Tips and tricks for development
+
+* It is good idea to use `IPFS_PUBLISH_CONFIG` env. variable to set custom config
+location for development.
+* If you want to see exceptions with stack-trace set `IPFS_PUBLISH_EXCEPTIONS` env. variable to `True`.

Dockerfile

@@ -18,7 +18,7 @@ WORKDIR /app
 
 RUN mkdir -p /data/ipfs_publish \
   && echo 'host = "localhost"\n\
-port = 8080\n\
+port = 8000\n\
 \n\
 [repos]\n\
 ' > $IPFS_PUBLISH_CONFIG
@@ -31,4 +31,4 @@ ENTRYPOINT ["./startup.sh"]
 CMD ["server"]
 
 # Http webhook server endpoint
-EXPOSE 8080
+EXPOSE 8000

README.md

@@ -31,6 +31,7 @@ CLI is in place to manage the repos.
 * Publish specific branch - you can specify which branch should be published from the repo
 * Build script - before adding to IPFS you can run script/binary inside the cloned repo
 * After publish script - after the publishing to IPFS, this script is run with argument of the created IPFS address
+* Direct DNSLink update for CloudFlare DNS provider
 
 ### Git providers
 
@@ -50,7 +51,7 @@ isolated from rest of your machine!**
 
 * Python 3.7 and higher
 * Git
-* go-ipfs daemon
+* go-ipfs daemon (tested with version 4.23)
 * UNIX-Like machine with public IP
 
 ### pip
@@ -74,7 +75,7 @@ version: '3'
 
 services:
   ipfs:
-    image: ipfs/go-ipfs:v0.4.18
+    image: ipfs/go-ipfs:v0.4.23
     volumes:
       - /data/ipfs # or you can mount it directly to some directory on your system
   ipfs-publish:

docker-compose.yaml

@@ -2,7 +2,7 @@ version: '3'
 
 services:
   ipfs:
-    image: ipfs/go-ipfs:v0.4.18
+    image: ipfs/go-ipfs:v0.4.23
     volumes:
       - /data/ipfs
   ipfs-publish:
@@ -14,8 +14,7 @@ services:
     environment:
       IPFS_PUBLISH_CONFIG: /data/ipfs_publish/config.toml
       IPFS_PUBLISH_VERBOSITY: 3
-      IPFS_PUBLISH_IPFS_HOST: ipfs
-      IPFS_PUBLISH_IPFS_PORT: 5001
+      IPFS_PUBLISH_IPFS_MULTIADDR: /dns4/ipfs/tcp/5001/http/
     volumes:
       - /data/ipfs_publish
     depends_on:

docs/index.md

@@ -49,7 +49,7 @@ webhook server is listening for incoming connections. And volume on path `/data/
 This image does not have IPFS daemon, therefore you have to provide connectivity to the daemon of your choice. 
 
 !!! info "go-ipfs verion"
-    ipfs-publish is tested with go-ipfs version **v0.4.18**, using different versions might result in unexpected behaviour!
+    ipfs-publish is tested with go-ipfs version **v0.4.23**, using different versions might result in unexpected behaviour!
 
 Easiest way to deploy ipfs-publish is using `docker-compose`, together with `go-ipfs` as container. 
 You can use this YAML configuration for it:
@@ -59,16 +59,15 @@ version: '3'
 
 services:
   ipfs:
-    image: ipfs/go-ipfs:v0.4.18
+    image: ipfs/go-ipfs:v0.4.23
     volumes:
       - /data/ipfs # or you can mount it directly to some directory on your system
   ipfs-publish:
     image: auhau/ipfs-publish
     environment:
       IPFS_PUBLISH_CONFIG: /data/ipfs_publish/config.toml
       IPFS_PUBLISH_VERBOSITY: 3
-      IPFS_PUBLISH_IPFS_HOST: ipfs
-      IPFS_PUBLISH_IPFS_PORT: 5001
+      IPFS_PUBLISH_IPFS_MULTIADDR: /dns4/ipfs/tcp/5001/http
     volumes:
       - /data/ipfs_publish
     depends_on:
@@ -82,7 +81,7 @@ If you have running IPFS daemon on the host like this:
 
 ```shell
 $ docker run -e IPFS_PUBLISH_CONFIG=/data/ipfs_publish/config.toml
- -e IPFS_PUBLISH_IPFS_HOST=localhost -e IPFS_PUBLISH_IPFS_PORT=5001 --network="host" auhau/ipfs_publish
+ -e IPFS_PUBLISH_IPFS_MULTIADDR=/ip4/127.0.0.1/tcp/5001/http --network="host" auhau/ipfs_publish
 ```
 
 !!! warning "Host network"
@@ -232,14 +231,15 @@ Running on http://localhost:8080 (CTRL + C to quit)
 When repo is being published it follows these steps:
 
 1. Freshly clone the Git repo into temporary directory, the default branch is checked out.
-2. If `build_bin` is defined, it is executed inside root of the repo.
-3. The `.git` folder is removed and if the `.ipfs_publish_ignore` file is present in root of the repo, the files 
+1. If `build_bin` is defined, it is executed inside root of the repo.
+1. The `.git` folder is removed and if the `.ipfs_publish_ignore` file is present in root of the repo, the files 
 specified in the file are removed.
-4. The old pinned version is unpinned.
-5. If `publish_dir` is specified, then this folder is added and pinned (if configured) to IPFS, otherwise the root of the repo is added.
-6. If publishing to IPNS is configured, the IPNS entry is updated.
-7. If `after_publish_bin` is defined, then it is executed inside root of the repo and the added IPFS hash is passed as argument.
-8. Cleanup of the repo.
+1. The old pinned version is unpinned.
+1. If `publish_dir` is specified, then this folder is added and pinned (if configured) to IPFS, otherwise the root of the repo is added.
+1. If publishing to IPNS is configured, the IPNS entry is updated.
+1. If CloudFlare DNS publishing is configured, then the latest CID is updated on configured DNS entry.
+1. If `after_publish_bin` is defined, then it is executed inside root of the repo and the added CID is passed as argument.
+1. Cleanup of the repo.
 
 ### Ignore files
 
@@ -266,7 +266,7 @@ build_bin = "jekyll build"
 ### After-publish binary
 
 Similarly to building binary, there is also support for running a command after publishing to the IPFS. This can be
-used for example to directly set the IPFS hash to your dns_link TXT record and not depend on IPNS. The published
+used for example to directly set the CID to your dns_link TXT record and not depend on IPNS. The published
 IPFS address is passed as a argument to the binary.
 
 The binary can be specified during the bootstrapping of the repo using CLI , or later on added into the config file under "execute" subsection
@@ -291,3 +291,32 @@ repo, or later on adding `branch=<name>` to the config:
 [repos.github_com_auhau_auhau_github_io]
 branch = "gh-pages"
 ```
+
+### CloudFlare
+
+As IPNS is currently not very performent for resolution, it is best practice to avoid it. In order to overcome this, there
+is native support for changing DNSLink DNS record on CloudFlare provider (for other providers you have to write your own
+script and use after-publish hook).
+
+In order for this to work, ipfs-publish has to have access to CloudFlare. You have to provide a API token, for all 
+possible ways how to do that see [python-cloudflare](https://github.com/cloudflare/python-cloudflare/#providing-cloudflare-username-and-api-key)
+documentation.
+
+!!! danger "DNS Access"
+    Configure this with security in mind! If somebody would stole your API token, he can very effectively attack your website!
+    
+!!! tip "Scoped API tokens"
+    Use API Tokens with smallest privileges (eq. edit DNS entry) and limit them only to Zone that is needed!
+    
+!!! success "Recommended setting"
+    It is recommended to use the environment variable `CF_API_KEY` with API Token, preferably configured on the systemd
+    unit as these files are not readable without `sudo` and the environment variables are not passed to any hooks 
+    (`build` and `after_publish` script), which should provide hopefully satisfying level of security. 
+
+If you want to add support for this later on, you have to specify Zone and DNS ID like so:
+
+```toml
+[repos.github_com_auhau_auhau_github_io.cloudflare]
+zone_id = "fb91814936c9812312aasdfc57ac516e98"
+dns_id = "c964dfc80ed523124d1casd513hu0a52"
+```

publish/cli.py

@@ -34,7 +34,6 @@ def entrypoint(args: typing.Sequence[str], obj: typing.Optional[dict] = None):
         exit(1)
 
 
-# TODO: Add --config to specify path to the config file
 @click.group()
 @click.option('--quiet', '-q', is_flag=True, help="Don't print anything")
 @click.option('--verbose', '-v', count=True, help="Prints additional info. More Vs, more info! (-vvv...)")

publish/cloudflare.py

@@ -0,0 +1,84 @@
+import logging
+import typing
+
+import CloudFlare
+import inquirer
+
+from publish import exceptions
+
+logger = logging.getLogger('publish.cloudflare')
+
+
+def bootstrap_cloudflare() -> typing.Tuple[typing.Optional[str], typing.Optional[str]]:
+    if not inquirer.shortcuts.confirm('Do you want to update DNSLink on Cloudflare?', default=True):
+        return None, None
+
+    cf = CloudFlare.CloudFlare()
+    try:
+        cf.user.tokens.verify()
+    except CloudFlare.exceptions.CloudFlareAPIError:
+        print('>>> You don\'t have configured CloudFlare token!')
+        print('>>> Either rerun this with proper configuration or specify ID of the Zone and TXT DNS entry which should be used for DNSLink.')
+        print('>>> If you need help with the CloudFlare configuration see: https://github.com/cloudflare/python-cloudflare')
+        zone_id = inquirer.shortcuts.text('Zone ID')
+        dns_id = inquirer.shortcuts.text('DNS entry ID')
+        return zone_id, dns_id
+
+    print('>>> Lets find the right record you will want to update.')
+
+    try:
+        available_zones = cf.zones.get()
+        zone_id = inquirer.shortcuts.list_input('In which zone should be the DNSLink edited?', choices=[(x['name'], x['id']) for x in available_zones])
+    except CloudFlare.exceptions.CloudFlareAPIError:
+        print('>>> Your token does not have sufficient rights to list zones!')
+        zone_id = inquirer.shortcuts.text('Please provide Zone ID where should DNSLink be edited')
+
+    if inquirer.shortcuts.confirm('Does the DNSLink TXT entry already exists?'):
+        dns_records = cf.zones.dns_records.get(zone_id, params={'type': 'TXT', 'per_page': 100})
+        dns_id = inquirer.shortcuts.list_input('Which entry you want to use?', choices=[(f'{x["name"]}: {x["content"][:40]}', x['id']) for x in dns_records])
+    else:
+        print('>>> Ok, lets create it then!')
+        dns_name = inquirer.shortcuts.text('Where it should be placed (eq. full domain name with subdomain, it should probably start with _dnslink)')
+        dns_id = cf.zones.dns_records.post(zone_id, data={'name': dns_name, 'type': 'TXT', 'content': 'dnslink='})["id"]
+        print(f'>>> Entry with ID {dns_id} created!')
+
+    return zone_id, dns_id
+
+
+# TODO: Verify that cf.user.tokens.verify() works with Email & Token
+# TODO: Verify that ENV configured token does not leak to scripts
+class CloudFlareMixin:
+
+    dns_id: typing.Optional[str] = None
+    """
+    DNS ID of TXT record where the DNSLink should be updated.
+    """
+
+    zone_id: typing.Optional[str] = None
+    """
+    Zone ID of the DNS record where it will be modified.
+    """
+
+    def __init__(self, dns_id: str = None, zone_id: str = None):
+        if (dns_id or zone_id) and not (dns_id and zone_id):
+            raise exceptions.ConfigException('You have to set both dns_id and zone_id! Only one does not make sense.')
+
+        self.cf = CloudFlare.CloudFlare()
+        self.dns_id = dns_id
+        self.zone_id = zone_id
+
+    def update_dns(self, cid: str):
+        if not self.dns_id or not self.zone_id:
+            raise exceptions.ConfigException('dns_id and zone_id not set. Not possible to update DNS!')
+
+        try:
+            self.cf.user.tokens.verify()
+        except CloudFlare.exceptions.CloudFlareAPIError:
+            raise exceptions.PublishingException('CloudFlare access not configured!')
+
+        logger.info('Publishing new CID to CloudFlare DNSLink')
+
+        record = self.cf.zones.dns_records.get(self.zone_id, self.dns_id)
+        record['content'] = f'dnslink={cid}'
+        self.cf.zones.dns_records.put(self.zone_id, self.dns_id, data=record)
+

publish/config.py

@@ -80,7 +80,7 @@ def ipfs(self):  # type: () -> ipfshttpclient.Client
                 port = os.environ.get(ENV_NAME_IPFS_PORT)
 
             # Hack to allow cross-platform Docker to reference the Docker host's machine with $HOST_ADDR
-            if host.startswith('$'):
+            if host and host.startswith('$'):
                 logger.info(f'Resolving host name from environment variable {host}')
                 host = os.environ[host[1:]]
 

publish/publishing.py

@@ -15,6 +15,7 @@
 import inquirer
 import ipfshttpclient
 
+from publish import cloudflare
 from publish import config as config_module, exceptions, PUBLISH_IGNORE_FILENAME, DEFAULT_LENGTH_OF_SECRET, \
     IPNS_KEYS_NAME_PREFIX, IPNS_KEYS_TYPE, helpers
 
@@ -222,7 +223,7 @@ def bootstrap_repo(config: config_module.Config, git_repo_url=None, **kwargs) ->
     return GenericRepo.bootstrap_repo(config, git_repo_url=git_repo_url, **kwargs)
 
 
-class GenericRepo:
+class GenericRepo(cloudflare.CloudFlareMixin):
     """
     Generic Repo's class that represent and store all information about Git repository that can be placed on any
     Git's provider.
@@ -244,6 +245,8 @@ class GenericRepo:
         'ipns_key': 'ipns',
         'ipns_addr': 'ipns',
         'ipns_lifetime': 'ipns',
+        'zone_id': 'cloudflare',
+        'dns_id': 'cloudflare'
     }
     """
     Mapping that maps the repo's properties into TOML's config sections.
@@ -315,7 +318,7 @@ def __init__(self, config: config_module.Config, name: str, git_repo_url: str, s
                  branch: typing.Optional[str] = None,
                  ipns_addr: typing.Optional[str] = None, ipns_key: typing.Optional[str] = None, ipns_lifetime='24h',
                  republish=False, pin=True, last_ipfs_addr=None, publish_dir: str = '/',
-                 build_bin=None, after_publish_bin=None, ipns_ttl='15m'):
+                 build_bin=None, after_publish_bin=None, ipns_ttl='15m', **kwargs):
         self.name = name
         self.git_repo_url = git_repo_url
         self.branch = branch
@@ -336,6 +339,8 @@ def __init__(self, config: config_module.Config, name: str, git_repo_url: str, s
         self.build_bin = build_bin
         self.after_publish_bin = after_publish_bin
 
+        super().__init__(**kwargs)
+
     @property
     def webhook_url(self) -> str:
         """
@@ -386,28 +391,34 @@ def publish_repo(self) -> None:
         publish_dir = path / (self.publish_dir[1:] if self.publish_dir.startswith('/') else self.publish_dir)
         logger.info(f'Adding directory {publish_dir} to IPFS')
         result = ipfs.add(publish_dir, recursive=True, pin=self.pin)
-        self.last_ipfs_addr = f'/ipfs/{result[-1]["Hash"]}/'
-        logger.info(f'Repo successfully added to IPFS with hash: {self.last_ipfs_addr}')
+        cid = f'/ipfs/{result[-1]["Hash"]}/'
+        self.last_ipfs_addr = cid
+        logger.info(f'Repo successfully added to IPFS with hash: {cid}')
 
         if self.ipns_key is not None:
-            self.publish_name()
+            self.publish_name(cid)
+
+        try:
+            self.update_dns(cid)
+        except exceptions.ConfigException:
+            pass
 
         if self.after_publish_bin:
-            self._run_bin(path, self.after_publish_bin, self.last_ipfs_addr)
+            self._run_bin(path, self.after_publish_bin, cid)
 
         self._cleanup_repo(path)
 
-    def publish_name(self) -> None:
+    def publish_name(self, cid) -> None:
         """
         Main method that handles publishing of the IPFS addr into IPNS.
         :return:
         """
-        if self.last_ipfs_addr is None:
+        if cid is None:
             return
 
         logger.info('Updating IPNS name')
         ipfs = self.config.ipfs
-        ipfs.name.publish(self.last_ipfs_addr, key=self.ipns_key, ttl=self.ipns_ttl)
+        ipfs.name.publish(cid, key=self.ipns_key, ttl=self.ipns_ttl)
         logger.info('IPNS successfully published')
 
     def _clone_repo(self) -> pathlib.Path:
@@ -562,6 +573,7 @@ def bootstrap_repo(cls, config: config_module.Config, name=None, git_repo_url=No
             branch = get_default_branch(git_repo_url)
 
         ipns_key, ipns_addr = bootstrap_ipns(config, name, ipns_key)
+        zone_id, dns_id = cloudflare.bootstrap_cloudflare()
 
         if secret is None:
             secret = ''.join(
@@ -592,15 +604,17 @@ def bootstrap_repo(cls, config: config_module.Config, name=None, git_repo_url=No
             raise exceptions.RepoException('Passed ttl is not valid! Supported units are: h(our), m(inute), '
                                            's(seconds)!')
 
-        if ipns_key is None and after_publish_bin is None:
+        if ipns_key is None and after_publish_bin is None and zone_id is None:
             raise exceptions.RepoException(
-                'You have choose not to use IPNS and you also have not specified any after publish command. '
-                'This does not make sense! What do you want to do with this setting?! I have no idea, so aborting!')
+                'You have choose not to use IPNS, not modify DNSLink entry on CloudFlare and you also have not '
+                'specified any after publish command. This does not make sense! What do you want to do '
+                'with this setting?! I have no idea, so aborting!')
 
         return cls(config=config, name=name, git_repo_url=git_repo_url, branch=branch, secret=secret, pin=pin,
                    publish_dir=publish_dir,
                    ipns_key=ipns_key, ipns_addr=ipns_addr, build_bin=build_bin, after_publish_bin=after_publish_bin,
-                   republish=republish, ipns_lifetime=ipns_lifetime, ipns_ttl=ipns_ttl)
+                   republish=republish, ipns_lifetime=ipns_lifetime, ipns_ttl=ipns_ttl, dns_id=dns_id,
+                   zone_id=zone_id)
 
 
 def bootstrap_ipns(config: config_module.Config, name: str, ipns_key: str = None) -> typing.Tuple[str, str]: