Merge branch 'dev' of https://github.com/Neilpang/acme.sh into dev

README.md

@@ -204,7 +204,7 @@ Install/copy the cert/key to the production Apache or Nginx path.
 The cert will be renewed every **60** days by default (which is configurable). Once the cert is renewed, the Apache/Nginx service will be reloaded automatically by the command: `service apache2 force-reload` or `service nginx force-reload`.
 
 
-**Please take care:  The reloadcmd is very important. The cert can be automatically renewed, but, without a correct 'reloadcmd' the cert may not be flushed to your server(like nginx or apache), then your website will not be able to show renewwed cert in 60 days.**
+**Please take care:  The reloadcmd is very important. The cert can be automatically renewed, but, without a correct 'reloadcmd' the cert may not be flushed to your server(like nginx or apache), then your website will not be able to show renewed cert in 60 days.**
 
 # 4. Use Standalone server to issue cert
 

deploy/README.md

@@ -31,7 +31,146 @@ acme.sh --deploy -d ftp.example.com --deploy-hook kong
 
 ## 3. Deploy the cert to remote server through SSH access
 
-(TODO)
+The ssh deploy plugin allows you to deploy certificates to a remote host
+using SSH command to connect to the remote server.  The ssh plugin is invoked
+with the following command...
+
+```sh
+acme.sh --deploy -d example.com --deploy-hook ssh
+```
+Prior to running this for the first time you must tell the plugin where
+and how to deploy the certificates.  This is done by exporting the following
+environment variables.  This is not required for subsequent runs as the
+values are stored by acme.sh in the domain configuration files.
+
+Required...
+```
+export DEPLOY_SSH_USER=username
+```
+Optional...
+```
+export DEPLOY_SSH_CMD=custom ssh command
+export DEPLOY_SSH_SERVER=url or ip address of remote host
+export DEPLOY_SSH_KEYFILE=filename for private key
+export DEPLOY_SSH_CERTFILE=filename for certificate file
+export DEPLOY_SSH_CAFILE=filename for intermediate CA file
+export DEPLOY_SSH_FULLCHAIN=filename for fullchain file
+export DEPLOY_SSH_REMOTE_CMD=command to execute on remote host
+export DEPLOY_SSH_BACKUP=yes or no
+```
+
+**DEPLOY_SSH_USER**
+Username at the remote host that SSH will login with. Note that
+SSH must be able to login to remote host without a password... SSH Keys
+must have been exchanged with the remote host. Validate and test that you
+can login to USER@URL from the host running acme.sh before using this script.
+
+The USER@URL at the remote server must also have has permissions to write to
+the target location of the certificate files and to execute any commands
+(e.g. to stop/start services).
+
+**DEPLOY_SSH_CMD**
+You can customize the ssh command used to connect to the remote host. For example
+if you need to connect to a specific port at the remote server you can set this
+to, for example, "ssh -p 22" or to use `sshpass` to provide password inline
+instead of exchanging ssh keys (this is not recommended, using keys is
+more secure).
+
+**DEPLOY_SSH_SERVER**
+URL or IP Address of the remote server.  If not provided then the domain
+name provided on the acme.sh --deploy command line is used.
+
+**DEPLOY_SSH_KEYFILE**
+Target filename for the private key issued by LetsEncrypt.
+
+**DEPLOY_SSH_CERTFILE**
+Target filename for the certificate issued by LetsEncrypt.
+If this is the same as the previous filename (for keyfile) then it is
+appended to the same file.
+
+**DEPLOY_SSH_CAFILE**
+Target filename for the CA intermediate certificate issued by LetsEncrypt.
+If this is the same as a previous filename (for keyfile or certfile) then
+it is appended to the same file.
+
+**DEPLOY_SSH_FULLCHAIN**
+Target filename for the fullchain certificate issued by LetsEncrypt.
+If this is the same as a previous filename (for keyfile, certfile or
+cafile) then it is appended to the same file.
+
+**DEPLOY_SSH_REMOTE_CMD**
+Command to execute on the remote server after copying any certificates.  This
+could be any additional command required for example to stop and restart
+the service.
+
+**DEPLOY_SSH_BACKUP**
+Before writing a certificate file to the remote server the existing
+certificate will be copied to a backup directory on the remote server.
+These are placed in a hidden directory in the home directory of the SSH
+user
+```sh
+~/.acme_ssh_deploy/[domain name]-backup-[timestamp]
+```
+Any backups older than 180 days will be deleted when new certificates
+are deployed.  This defaults to "yes" set to "no" to disable backup.
+
+###Eamples using SSH deploy
+The following example illustrates deploying certifcates to a QNAP NAS
+(tested with QTS version 4.2.3)
+
+```sh
+export DEPLOY_SSH_USER="admin"
+export DEPLOY_SSH_KEYFILE="/etc/stunnel/stunnel.pem"
+export DEPLOY_SSH_CERTFILE="/etc/stunnel/stunnel.pem"
+export DEPLOY_SSH_CAFILE="/etc/stunnel/uca.pem"
+export DEPLOY_SSH_REMOTE_CMD="/etc/init.d/stunnel.sh restart"
+
+acme.sh --deploy -d qnap.example.com --deploy-hook ssh
+```
+Note how in this example both the private key and certificate point to
+the same file.  This will result in the certificate being appended
+to the same file as the private key... a common requirement of several
+services.
+
+The next example illustates deploying certificates to a Unifi
+Contolller (tested with version 5.4.11).
+
+```sh
+export DEPLOY_SSH_USER="root"
+export DEPLOY_SSH_KEYFILE="/var/lib/unifi/unifi.example.com.key"
+export DEPLOY_SSH_FULLCHAIN="/var/lib/unifi/unifi.example.com.cer"
+export DEPLOY_SSH_REMOTE_CMD="openssl pkcs12 -export \
+   -inkey /var/lib/unifi/unifi.example.com.key \
+   -in /var/lib/unifi/unifi.example.com.cer \
+   -out /var/lib/unifi/unifi.example.com.p12 \
+   -name ubnt -password pass:temppass \
+ && keytool -importkeystore -deststorepass aircontrolenterprise \
+   -destkeypass aircontrolenterprise \
+   -destkeystore /var/lib/unifi/keystore \
+   -srckeystore /var/lib/unifi/unifi.example.com.p12 \
+   -srcstoretype PKCS12 -srcstorepass temppass -alias ubnt -noprompt \
+ && service unifi restart"
+
+acme.sh --deploy -d unifi.example.com --deploy-hook ssh
+```
+In this exmple we execute several commands on the remote host
+after the certificate files have been copied... to generate a pkcs12 file
+compatible with Unifi, to import it into the Unifi keystore and then finaly
+to restart the service.
+
+Note also that once the certificate is imported
+into the keystore the individual certificate files are no longer
+required. We could if we desired delete those files immediately. If we
+do that then we should disable backup at the remote host (as there are
+no files to backup -- they were erased during deployment). For example...
+```sh
+export DEPLOY_SSH_BACKUP=no
+# modify the end of the remote command...
+&& rm /var/lib/unifi/unifi.example.com.key \
+      /var/lib/unifi/unifi.example.com.cer \
+      /var/lib/unifi/unifi.example.com.p12 \
+&& service unifi restart
+```
 
 ## 4. Deploy the cert to local vsftpd server
 

deploy/ssh.sh

@@ -0,0 +1,205 @@
+#!/usr/bin/env sh
+
+# Script to deploy certificates to remote server by SSH
+# Note that SSH must be able to login to remote host without a password...
+# SSH Keys must have been exchanged with the remote host.  Validate and
+# test that you can login to USER@SERVER from the host running acme.sh before
+# using this script.
+#
+# The following variables exported from environment will be used.
+# If not set then values previously saved in domain.conf file are used.
+#
+# Only a username is required.  All others are optional.
+#
+# The following examples are for QNAP NAS running QTS 4.2 
+# export DEPLOY_SSH_CMD=""  # defaults to ssh
+# export DEPLOY_SSH_USER="admin"  # required
+# export DEPLOY_SSH_SERVER="qnap"  # defaults to domain name
+# export DEPLOY_SSH_KEYFILE="/etc/stunnel/stunnel.pem"
+# export DEPLOY_SSH_CERTFILE="/etc/stunnel/stunnel.pem"
+# export DEPLOY_SSH_CAFILE="/etc/stunnel/uca.pem"
+# export DEPLOY_SSH_FULLCHAIN=""
+# export DEPLOY_SSH_REMOTE_CMD="/etc/init.d/stunnel.sh restart"
+# export DEPLOY_SSH_BACKUP=""  # yes or no, default to yes
+#
+########  Public functions #####################
+
+#domain keyfile certfile cafile fullchain
+ssh_deploy() {
+  _cdomain="$1"
+  _ckey="$2"
+  _ccert="$3"
+  _cca="$4"
+  _cfullchain="$5"
+  _cmdstr=""
+  _homedir='~'
+  _backupprefix="$_homedir/.acme_ssh_deploy/$_cdomain-backup"
+  _backupdir="$_backupprefix-$(_utc_date | tr ' ' '-')"
+
+  if [ -f "$DOMAIN_CONF" ]; then
+    # shellcheck disable=SC1090
+    . "$DOMAIN_CONF"
+  fi
+
+  _debug _cdomain "$_cdomain"
+  _debug _ckey "$_ckey"
+  _debug _ccert "$_ccert"
+  _debug _cca "$_cca"
+  _debug _cfullchain "$_cfullchain"
+
+  # USER is required to login by SSH to remote host.
+  if [ -z "$DEPLOY_SSH_USER" ]; then
+    if [ -z "$Le_Deploy_ssh_user" ]; then
+      _err "DEPLOY_SSH_USER not defined."
+      return 1
+    fi
+  else
+    Le_Deploy_ssh_user="$DEPLOY_SSH_USER"
+    _savedomainconf Le_Deploy_ssh_user "$Le_Deploy_ssh_user"
+  fi
+
+  # SERVER is optional. If not provided then use _cdomain
+  if [ -n "$DEPLOY_SSH_SERVER" ]; then
+    Le_Deploy_ssh_server="$DEPLOY_SSH_SERVER"
+    _savedomainconf Le_Deploy_ssh_server "$Le_Deploy_ssh_server"
+  elif [ -z "$Le_Deploy_ssh_server" ]; then
+    Le_Deploy_ssh_server="$_cdomain"
+  fi
+
+  # CMD is optional. If not provided then use ssh
+  if [ -n "$DEPLOY_SSH_CMD" ]; then
+    Le_Deploy_ssh_cmd="$DEPLOY_SSH_CMD"
+    _savedomainconf Le_Deploy_ssh_cmd "$Le_Deploy_ssh_cmd"
+  elif [ -z "$Le_Deploy_ssh_cmd" ]; then
+    Le_Deploy_ssh_cmd="ssh"
+  fi
+
+  # BACKUP is optional. If not provided then default to yes
+  if [ "$DEPLOY_SSH_BACKUP" = "no" ]; then
+    Le_Deploy_ssh_backup="no"
+  elif [ -z "$Le_Deploy_ssh_backup" ]; then
+    Le_Deploy_ssh_backup="yes"
+  fi
+  _savedomainconf Le_Deploy_ssh_backup "$Le_Deploy_ssh_backup"
+
+  _info "Deploy certificates to remote server $Le_Deploy_ssh_user@$Le_Deploy_ssh_server"
+
+  # KEYFILE is optional.
+  # If provided then private key will be copied to provided filename.
+  if [ -n "$DEPLOY_SSH_KEYFILE" ]; then
+    Le_Deploy_ssh_keyfile="$DEPLOY_SSH_KEYFILE"
+    _savedomainconf Le_Deploy_ssh_keyfile "$Le_Deploy_ssh_keyfile"
+  fi
+  if [ -n "$Le_Deploy_ssh_keyfile" ]; then
+    if [ "$Le_Deploy_ssh_backup" = "yes" ]; then
+      # backup file we are about to overwrite.
+      _cmdstr="$_cmdstr cp $Le_Deploy_ssh_keyfile $_backupdir >/dev/null;"
+    fi
+    # copy new certificate into file.
+    _cmdstr="$_cmdstr echo \"$(cat "$_ckey")\" > $Le_Deploy_ssh_keyfile;"
+    _info "will copy private key to remote file $Le_Deploy_ssh_keyfile"
+  fi
+
+  # CERTFILE is optional.
+  # If provided then private key will be copied or appended to provided filename.
+  if [ -n "$DEPLOY_SSH_CERTFILE" ]; then
+    Le_Deploy_ssh_certfile="$DEPLOY_SSH_CERTFILE"
+    _savedomainconf Le_Deploy_ssh_certfile "$Le_Deploy_ssh_certfile"
+  fi
+  if [ -n "$Le_Deploy_ssh_certfile" ]; then
+    _pipe=">"
+    if [ "$Le_Deploy_ssh_certfile" = "$Le_Deploy_ssh_keyfile" ]; then
+      # if filename is same as previous file then append.
+      _pipe=">>"
+    elif [ "$Le_Deploy_ssh_backup" = "yes" ]; then
+      # backup file we are about to overwrite.
+      _cmdstr="$_cmdstr cp $Le_Deploy_ssh_certfile $_backupdir >/dev/null;"
+    fi
+    # copy new certificate into file.
+    _cmdstr="$_cmdstr echo \"$(cat "$_ccert")\" $_pipe $Le_Deploy_ssh_certfile;"
+    _info "will copy certificate to remote file $Le_Deploy_ssh_certfile"
+  fi
+
+  # CAFILE is optional.
+  # If provided then CA intermediate certificate will be copied or appended to provided filename.
+  if [ -n "$DEPLOY_SSH_CAFILE" ]; then
+    Le_Deploy_ssh_cafile="$DEPLOY_SSH_CAFILE"
+    _savedomainconf Le_Deploy_ssh_cafile "$Le_Deploy_ssh_cafile"
+  fi
+  if [ -n "$Le_Deploy_ssh_cafile" ]; then
+    _pipe=">"
+    if [ "$Le_Deploy_ssh_cafile" = "$Le_Deploy_ssh_keyfile" ] \
+      || [ "$Le_Deploy_ssh_cafile" = "$Le_Deploy_ssh_certfile" ]; then
+      # if filename is same as previous file then append.
+      _pipe=">>"
+    elif [ "$Le_Deploy_ssh_backup" = "yes" ]; then
+      # backup file we are about to overwrite.
+      _cmdstr="$_cmdstr cp $Le_Deploy_ssh_cafile $_backupdir >/dev/null;"
+    fi
+    # copy new certificate into file.
+    _cmdstr="$_cmdstr echo \"$(cat "$_cca")\" $_pipe $Le_Deploy_ssh_cafile;"
+    _info "will copy CA file to remote file $Le_Deploy_ssh_cafile"
+  fi
+
+  # FULLCHAIN is optional.
+  # If provided then fullchain certificate will be copied or appended to provided filename.
+  if [ -n "$DEPLOY_SSH_FULLCHAIN" ]; then
+    Le_Deploy_ssh_fullchain="$DEPLOY_SSH_FULLCHAIN"
+    _savedomainconf Le_Deploy_ssh_fullchain "$Le_Deploy_ssh_fullchain"
+  fi
+  if [ -n "$Le_Deploy_ssh_fullchain" ]; then
+    _pipe=">"
+    if [ "$Le_Deploy_ssh_fullchain" = "$Le_Deploy_ssh_keyfile" ] \
+      || [ "$Le_Deploy_ssh_fullchain" = "$Le_Deploy_ssh_certfile" ] \
+      || [ "$Le_Deploy_ssh_fullchain" = "$Le_Deploy_ssh_cafile" ]; then
+      # if filename is same as previous file then append.
+      _pipe=">>"
+    elif [ "$Le_Deploy_ssh_backup" = "yes" ]; then
+      # backup file we are about to overwrite.
+      _cmdstr="$_cmdstr cp $Le_Deploy_ssh_fullchain $_backupdir >/dev/null;"
+    fi
+    # copy new certificate into file.
+    _cmdstr="$_cmdstr echo \"$(cat "$_cfullchain")\" $_pipe $Le_Deploy_ssh_fullchain;"
+    _info "will copy fullchain to remote file $Le_Deploy_ssh_fullchain"
+  fi
+
+  # REMOTE_CMD is optional.
+  # If provided then this command will be executed on remote host.
+  if [ -n "$DEPLOY_SSH_REMOTE_CMD" ]; then
+    Le_Deploy_ssh_remote_cmd="$DEPLOY_SSH_REMOTE_CMD"
+    _savedomainconf Le_Deploy_ssh_remote_cmd "$Le_Deploy_ssh_remote_cmd"
+  fi
+  if [ -n "$Le_Deploy_ssh_remote_cmd" ]; then
+    _cmdstr="$_cmdstr $Le_Deploy_ssh_remote_cmd;"
+    _info "Will execute remote command $Le_Deploy_ssh_remote_cmd"
+  fi
+
+  if [ -z "$_cmdstr" ]; then
+    _err "No remote commands to excute. Failed to deploy certificates to remote server"
+    return 1
+  elif [ "$Le_Deploy_ssh_backup" = "yes" ]; then
+    # run cleanup on the backup directory, erase all older
+    # than 180 days (15552000 seconds).
+    _cmdstr="{ now=\"\$(date -u +%s)\"; for fn in $_backupprefix*; \
+do if [ -d \"\$fn\" ] && [ \"\$(expr \$now - \$(date -ur \$fn +%s) )\" -ge \"15552000\" ]; \
+then rm -rf \"\$fn\"; echo \"Backup \$fn deleted as older than 180 days\"; fi; done; }; $_cmdstr"
+    # Alternate version of above... _cmdstr="find $_backupprefix* -type d -mtime +180 2>/dev/null | xargs rm -rf; $_cmdstr"
+    # Create our backup directory for overwritten cert files.
+    _cmdstr="mkdir -p $_backupdir; $_cmdstr"
+    _info "Backup of old certificate files will be placed in remote directory $_backupdir"
+    _info "Backup directories erased after 180 days."
+  fi
+
+  _debug "Remote commands to execute: $_cmdstr"
+  _info "Submitting sequence of commands to remote server by ssh"
+  # quotations in bash cmd below intended.  Squash travis spellcheck error
+  # shellcheck disable=SC2029
+  $Le_Deploy_ssh_cmd -T "$Le_Deploy_ssh_user@$Le_Deploy_ssh_server" sh -c "'$_cmdstr'"
+  _ret="$?"
+
+  if [ "$_ret" != "0" ]; then
+    _err "Error code $_ret returned from $Le_Deploy_ssh_cmd"
+  fi
+
+  return $_ret
+}

dnsapi/dns_aws.sh

@@ -23,7 +23,7 @@ dns_aws_add() {
   AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY:-$(_readaccountconf_mutable AWS_SECRET_ACCESS_KEY)}"
 
   if [ -z "$AWS_ACCESS_KEY_ID" ] || [ -z "$AWS_SECRET_ACCESS_KEY" ]; then
-    _use_instance_role
+    _use_container_role || _use_instance_role
   fi
 
   if [ -z "$AWS_ACCESS_KEY_ID" ] || [ -z "$AWS_SECRET_ACCESS_KEY" ]; then
@@ -87,7 +87,7 @@ dns_aws_rm() {
   AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY:-$(_readaccountconf_mutable AWS_SECRET_ACCESS_KEY)}"
 
   if [ -z "$AWS_ACCESS_KEY_ID" ] || [ -z "$AWS_SECRET_ACCESS_KEY" ]; then
-    _use_instance_role
+    _use_container_role || _use_instance_role
   fi
 
   _debug "First detect the root zone"
@@ -174,17 +174,30 @@ _get_root() {
   return 1
 }
 
+_use_container_role() {
+  # automatically set if running inside ECS
+  if [ -z "$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI" ]; then
+    _debug "No ECS environment variable detected"
+    return 1
+  fi
+  _use_metadata "169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI"
+}
+
 _use_instance_role() {
   _url="http://169.254.169.254/latest/meta-data/iam/security-credentials/"
   _debug "_url" "$_url"
   if ! _get "$_url" true 1 | _head_n 1 | grep -Fq 200; then
-    _err "Unable to fetch IAM role from AWS instance metadata."
-    return
+    _debug "Unable to fetch IAM role from instance metadata"
+    return 1
   fi
   _aws_role=$(_get "$_url" "" 1)
   _debug "_aws_role" "$_aws_role"
+  _use_metadata "$_url$_aws_role"
+}
+
+_use_metadata() {
   _aws_creds="$(
-    _get "$_url$_aws_role" "" 1 \
+    _get "$1" "" 1 \
       | _normalizeJson \
       | tr '{,}' '\n' \
       | while read -r _line; do
@@ -201,6 +214,11 @@ _use_instance_role() {
         | paste -sd' ' -
   )"
   _secure_debug "_aws_creds" "$_aws_creds"
+
+  if [ -z "$_aws_creds" ]; then
+    return 1
+  fi
+
   eval "$_aws_creds"
   _using_role=true
 }

dnsapi/dns_azure.sh

@@ -1,5 +1,7 @@
 #!/usr/bin/env sh
 
+WIKI="https://github.com/Neilpang/acme.sh/wiki/How-to-use-Azure-DNS"
+
 ########  Public functions #####################
 
 # Usage: add  _acme-challenge.www.domain.com   "XKrxpRBosdIKFzxW_CT3KLZNf6q0HG9i01zxXp5CPBs"
@@ -69,12 +71,36 @@ dns_azure_add() {
 
   acmeRecordURI="https://management.azure.com$(printf '%s' "$_domain_id" | sed 's/\\//g')/TXT/$_sub_domain?api-version=2017-09-01"
   _debug "$acmeRecordURI"
-  body="{\"properties\": {\"TTL\": 3600, \"TXTRecords\": [{\"value\": [\"$txtvalue\"]}]}}"
+  # Get existing TXT record
+  _azure_rest GET "$acmeRecordURI" "" "$accesstoken"
+  values="{\"value\":[\"$txtvalue\"]}"
+  timestamp="$(_time)"
+  if [ "$_code" = "200" ]; then
+    vlist="$(echo "$response" | _egrep_o "\"value\"\s*:\s*\[\s*\"[^\"]*\"\s*]" | cut -d : -f 2 | tr -d "[]\"")"
+    _debug "existing TXT found"
+    _debug "$vlist"
+    existingts="$(echo "$response" | _egrep_o "\"acmetscheck\"\s*:\s*\"[^\"]*\"" | _head_n 1 | cut -d : -f 2 | tr -d "\"")"
+    if [ -z "$existingts" ]; then
+      # the record was not created by acme.sh. Copy the exisiting entires
+      existingts=$timestamp
+    fi
+    _diff="$(_math "$timestamp - $existingts")"
+    _debug "existing txt age: $_diff"
+    # only use recently added records and discard if older than 2 hours because they are probably orphaned
+    if [ "$_diff" -lt 7200 ]; then
+      _debug "existing txt value: $vlist"
+      for v in $vlist; do
+        values="$values ,{\"value\":[\"$v\"]}"
+      done
+    fi
+  fi
+  # Add the txtvalue TXT Record
+  body="{\"properties\":{\"metadata\":{\"acmetscheck\":\"$timestamp\"},\"TTL\":10, \"TXTRecords\":[$values]}}"
   _azure_rest PUT "$acmeRecordURI" "$body" "$accesstoken"
   if [ "$_code" = "200" ] || [ "$_code" = '201' ]; then
-    _info "validation record added"
+    _info "validation value added"
   else
-    _err "error adding validation record ($_code)"
+    _err "error adding validation value ($_code)"
     return 1
   fi
 }
@@ -141,13 +167,38 @@ dns_azure_rm() {
 
   acmeRecordURI="https://management.azure.com$(printf '%s' "$_domain_id" | sed 's/\\//g')/TXT/$_sub_domain?api-version=2017-09-01"
   _debug "$acmeRecordURI"
-  body="{\"properties\": {\"TTL\": 3600, \"TXTRecords\": [{\"value\": [\"$txtvalue\"]}]}}"
-  _azure_rest DELETE "$acmeRecordURI" "" "$accesstoken"
-  if [ "$_code" = "200" ] || [ "$_code" = '204' ]; then
-    _info "validation record removed"
-  else
-    _err "error removing validation record ($_code)"
-    return 1
+  # Get existing TXT record
+  _azure_rest GET "$acmeRecordURI" "" "$accesstoken"
+  timestamp="$(_time)"
+  if [ "$_code" = "200" ]; then
+    vlist="$(echo "$response" | _egrep_o "\"value\"\s*:\s*\[\s*\"[^\"]*\"\s*]" | cut -d : -f 2 | tr -d "[]\"" | grep -v "$txtvalue")"
+    values=""
+    comma=""
+    for v in $vlist; do
+      values="$values$comma{\"value\":[\"$v\"]}"
+      comma=","
+    done
+    if [ -z "$values" ]; then
+      # No values left remove record
+      _debug "removing validation record completely $acmeRecordURI"
+      _azure_rest DELETE "$acmeRecordURI" "" "$accesstoken"
+      if [ "$_code" = "200" ] || [ "$_code" = '204' ]; then
+        _info "validation record removed"
+      else
+        _err "error removing validation record ($_code)"
+        return 1
+      fi
+    else
+      # Remove only txtvalue from the TXT Record
+      body="{\"properties\":{\"metadata\":{\"acmetscheck\":\"$timestamp\"},\"TTL\":10, \"TXTRecords\":[$values]}}"
+      _azure_rest PUT "$acmeRecordURI" "$body" "$accesstoken"
+      if [ "$_code" = "200" ] || [ "$_code" = '201' ]; then
+        _info "validation value removed"
+      else
+        _err "error removing validation value ($_code)"
+        return 1
+      fi
+    fi
   fi
 }
 
@@ -159,52 +210,92 @@ _azure_rest() {
   data="$3"
   accesstoken="$4"
 
-  export _H1="authorization: Bearer $accesstoken"
-  export _H2="accept: application/json"
-  export _H3="Content-Type: application/json"
-
-  _debug "$ep"
-  if [ "$m" != "GET" ]; then
-    _debug data "$data"
-    response="$(_post "$data" "$ep" "" "$m")"
-  else
-    response="$(_get "$ep")"
-  fi
-  _debug2 response "$response"
-
-  _code="$(grep "^HTTP" "$HTTP_HEADER" | _tail_n 1 | cut -d " " -f 2 | tr -d "\r\n")"
-  _debug2 "http response code $_code"
-
-  if [ "$?" != "0" ]; then
-    _err "error $ep"
+  MAX_REQUEST_RETRY_TIMES=5
+  _request_retry_times=0
+  while [ "${_request_retry_times}" -lt "$MAX_REQUEST_RETRY_TIMES" ]; do
+    _debug3 _request_retry_times "$_request_retry_times"
+    export _H1="authorization: Bearer $accesstoken"
+    export _H2="accept: application/json"
+    export _H3="Content-Type: application/json"
+    # clear headers from previous request to avoid getting wrong http code on timeouts
+    :>"$HTTP_HEADER"
+    _debug "$ep"
+    if [ "$m" != "GET" ]; then
+      _secure_debug2 "data $data"
+      response="$(_post "$data" "$ep" "" "$m")"
+    else
+      response="$(_get "$ep")"
+    fi
+    _secure_debug2 "response $response"
+    _code="$(grep "^HTTP" "$HTTP_HEADER" | _tail_n 1 | cut -d " " -f 2 | tr -d "\r\n")"
+    _debug "http response code $_code"
+    if [ "$_code" = "401" ]; then
+      # we have an invalid access token set to expired
+      _saveaccountconf_mutable AZUREDNS_TOKENVALIDTO "0"
+      _err "access denied make sure your Azure settings are correct. See $WIKI"
+      return 1
+    fi
+    # See https://docs.microsoft.com/en-us/azure/architecture/best-practices/retry-service-specific#general-rest-and-retry-guidelines for retryable HTTP codes
+    if [ "$?" != "0" ] || [ -z "$_code" ] || [ "$_code" = "408" ] || [ "$_code" = "500" ] || [ "$_code" = "503" ] || [ "$_code" = "504" ]; then
+      _request_retry_times="$(_math "$_request_retry_times" + 1)"
+      _info "REST call error $_code retrying $ep in $_request_retry_times s"
+      _sleep "$_request_retry_times"
+      continue
+    fi
+    break
+  done
+  if [ "$_request_retry_times" = "$MAX_REQUEST_RETRY_TIMES" ]; then
+    _err "Error Azure REST called was retried $MAX_REQUEST_RETRY_TIMES times."
+    _err "Calling $ep failed."
     return 1
   fi
+  response="$(echo "$response" | _normalizeJson)"
   return 0
 }
 
 ## Ref: https://docs.microsoft.com/en-us/azure/active-directory/develop/active-directory-protocols-oauth-service-to-service#request-an-access-token
 _azure_getaccess_token() {
-  TENANTID=$1
+  tenantID=$1
   clientID=$2
   clientSecret=$3
 
+  accesstoken="${AZUREDNS_BEARERTOKEN:-$(_readaccountconf_mutable AZUREDNS_BEARERTOKEN)}"
+  expires_on="${AZUREDNS_TOKENVALIDTO:-$(_readaccountconf_mutable AZUREDNS_TOKENVALIDTO)}"
+
+  # can we reuse the bearer token?
+  if [ -n "$accesstoken" ] && [ -n "$expires_on" ]; then
+    if [ "$(_time)" -lt "$expires_on" ]; then
+      # brearer token is still valid - reuse it
+      _debug "reusing bearer token"
+      printf "%s" "$accesstoken"
+      return 0
+    else
+      _debug "bearer token expired"
+    fi
+  fi
+  _debug "getting new bearer token"
+
   export _H1="accept: application/json"
   export _H2="Content-Type: application/x-www-form-urlencoded"
 
   body="resource=$(printf "%s" 'https://management.core.windows.net/' | _url_encode)&client_id=$(printf "%s" "$clientID" | _url_encode)&client_secret=$(printf "%s" "$clientSecret" | _url_encode)&grant_type=client_credentials"
-  _debug data "$body"
-  response="$(_post "$body" "https://login.windows.net/$TENANTID/oauth2/token" "" "POST")"
+  _secure_debug2 "data $body"
+  response="$(_post "$body" "https://login.microsoftonline.com/$tenantID/oauth2/token" "" "POST")"
+  _secure_debug2 "response $response"
+  response="$(echo "$response" | _normalizeJson)"
   accesstoken=$(echo "$response" | _egrep_o "\"access_token\":\"[^\"]*\"" | _head_n 1 | cut -d : -f 2 | tr -d \")
-  _debug2 "response $response"
+  expires_on=$(echo "$response" | _egrep_o "\"expires_on\":\"[^\"]*\"" | _head_n 1 | cut -d : -f 2 | tr -d \")
 
   if [ -z "$accesstoken" ]; then
-    _err "no acccess token received"
+    _err "no acccess token received. Check your Azure settings see $WIKI"
     return 1
   fi
   if [ "$?" != "0" ]; then
     _err "error $response"
     return 1
   fi
+  _saveaccountconf_mutable AZUREDNS_BEARERTOKEN "$accesstoken"
+  _saveaccountconf_mutable AZUREDNS_TOKENVALIDTO "$expires_on"
   printf "%s" "$accesstoken"
   return 0
 }
@@ -222,7 +313,6 @@ _get_root() {
   ## Per https://docs.microsoft.com/en-us/azure/azure-subscription-service-limits#dns-limits you are limited to 100 Zone/subscriptions anyways
   ##
   _azure_rest GET "https://management.azure.com/subscriptions/$subscriptionId/providers/Microsoft.Network/dnszones?api-version=2017-09-01" "" "$accesstoken"
-
   # Find matching domain name is Json response
   while true; do
     h=$(printf "%s" "$domain" | cut -d . -f $i-100)

dnsapi/dns_cloudns.sh

@@ -26,30 +26,18 @@ dns_cloudns_add() {
 
   host="$(echo "$1" | sed "s/\.$zone\$//")"
   record=$2
-  record_id=$(_dns_cloudns_get_record_id "$zone" "$host")
 
   _debug zone "$zone"
   _debug host "$host"
   _debug record "$record"
-  _debug record_id "$record_id"
 
-  if [ -z "$record_id" ]; then
-    _info "Adding the TXT record for $1"
-    _dns_cloudns_http_api_call "dns/add-record.json" "domain-name=$zone&record-type=TXT&host=$host&record=$record&ttl=60"
-    if ! _contains "$response" "\"status\":\"Success\""; then
-      _err "Record cannot be added."
-      return 1
-    fi
-    _info "Added."
-  else
-    _info "Updating the TXT record for $1"
-    _dns_cloudns_http_api_call "dns/mod-record.json" "domain-name=$zone&record-id=$record_id&record-type=TXT&host=$host&record=$record&ttl=60"
-    if ! _contains "$response" "\"status\":\"Success\""; then
-      _err "The TXT record for $1 cannot be updated."
-      return 1
-    fi
-    _info "Updated."
+  _info "Adding the TXT record for $1"
+  _dns_cloudns_http_api_call "dns/add-record.json" "domain-name=$zone&record-type=TXT&host=$host&record=$record&ttl=60"
+  if ! _contains "$response" "\"status\":\"Success\""; then
+    _err "Record cannot be added."
+    return 1
   fi
+  _info "Added."
 
   return 0
 }
@@ -72,22 +60,32 @@ dns_cloudns_rm() {
 
   host="$(echo "$1" | sed "s/\.$zone\$//")"
   record=$2
-  record_id=$(_dns_cloudns_get_record_id "$zone" "$host")
 
-  _debug zone "$zone"
-  _debug host "$host"
-  _debug record "$record"
-  _debug record_id "$record_id"
+  _dns_cloudns_http_api_call "dns/records.json" "domain-name=$zone&host=$host&type=TXT"
+  if ! _contains "$response" "\"id\":"; then
+    return 1
+  fi
 
-  if [ ! -z "$record_id" ]; then
-    _info "Deleting the TXT record for $1"
-    _dns_cloudns_http_api_call "dns/delete-record.json" "domain-name=$zone&record-id=$record_id"
-    if ! _contains "$response" "\"status\":\"Success\""; then
-      _err "The TXT record for $1 cannot be deleted."
-      return 1
+  for i in $(echo "$response" | tr '{' "\n" | grep "$record"); do
+    record_id=$(echo "$i" | tr ',' "\n" | grep -E '^"id"' | sed -re 's/^\"id\"\:\"([0-9]+)\"$/\1/g')
+
+    if [ ! -z "$record_id" ]; then
+      _debug zone "$zone"
+      _debug host "$host"
+      _debug record "$record"
+      _debug record_id "$record_id"
+
+      _info "Deleting the TXT record for $1"
+      _dns_cloudns_http_api_call "dns/delete-record.json" "domain-name=$zone&record-id=$record_id"
+
+      if ! _contains "$response" "\"status\":\"Success\""; then
+        _err "The TXT record for $1 cannot be deleted."
+      else
+        _info "Deleted."
+      fi
     fi
-    _info "Deleted."
-  fi
+  done
+
   return 0
 }
 
@@ -126,7 +124,7 @@ _dns_cloudns_init_check() {
     return 1
   fi
 
-  #save the api id and password to the account conf file.
+  # save the api id and password to the account conf file.
   _saveaccountconf_mutable CLOUDNS_AUTH_ID "$CLOUDNS_AUTH_ID"
   _saveaccountconf_mutable CLOUDNS_SUB_AUTH_ID "$CLOUDNS_SUB_AUTH_ID"
   _saveaccountconf_mutable CLOUDNS_AUTH_PASSWORD "$CLOUDNS_AUTH_PASSWORD"
@@ -159,15 +157,6 @@ _dns_cloudns_get_zone_name() {
   return 1
 }
 
-_dns_cloudns_get_record_id() {
-  _dns_cloudns_http_api_call "dns/records.json" "domain-name=$1&host=$2&type=TXT"
-  if _contains "$response" "\"id\":"; then
-    echo "$response" | cut -d '"' -f 2
-    return 0
-  fi
-  return 1
-}
-
 _dns_cloudns_http_api_call() {
   method=$1
 
@@ -189,7 +178,7 @@ _dns_cloudns_http_api_call() {
 
   response="$(_get "$CLOUDNS_API/$method?$data")"
 
-  _debug2 response "$response"
+  _debug response "$response"
 
   return 0
 }

dnsapi/dns_me.sh

@@ -43,34 +43,16 @@ dns_me_add() {
     return 1
   fi
 
-  count=$(printf "%s\n" "$response" | _egrep_o "\"totalRecords\":[^,]*" | cut -d : -f 2)
-  _debug count "$count"
-  if [ "$count" = "0" ]; then
-    _info "Adding record"
-    if _me_rest POST "$_domain_id/records/" "{\"type\":\"TXT\",\"name\":\"$_sub_domain\",\"value\":\"$txtvalue\",\"gtdLocation\":\"DEFAULT\",\"ttl\":120}"; then
-      if printf -- "%s" "$response" | grep \"id\": >/dev/null; then
-        _info "Added"
-        #todo: check if the record takes effect
-        return 0
-      else
-        _err "Add txt record error."
-        return 1
-      fi
-    fi
-    _err "Add txt record error."
-  else
-    _info "Updating record"
-    record_id=$(printf "%s\n" "$response" | _egrep_o "\"id\":[^,]*" | cut -d : -f 2 | head -n 1)
-    _debug "record_id" "$record_id"
-
-    _me_rest PUT "$_domain_id/records/$record_id/" "{\"id\":\"$record_id\",\"type\":\"TXT\",\"name\":\"$_sub_domain\",\"value\":\"$txtvalue\",\"gtdLocation\":\"DEFAULT\",\"ttl\":120}"
-    if [ "$?" = "0" ]; then
-      _info "Updated"
+  _info "Adding record"
+  if _me_rest POST "$_domain_id/records/" "{\"type\":\"TXT\",\"name\":\"$_sub_domain\",\"value\":\"$txtvalue\",\"gtdLocation\":\"DEFAULT\",\"ttl\":120}"; then
+    if printf -- "%s" "$response" | grep \"id\": >/dev/null; then
+      _info "Added"
       #todo: check if the record takes effect
       return 0
+    else
+      _err "Add txt record error."
+      return 1
     fi
-    _err "Update error"
-    return 1
   fi
 
 }
@@ -96,7 +78,7 @@ dns_me_rm() {
   if [ "$count" = "0" ]; then
     _info "Don't need to remove."
   else
-    record_id=$(printf "%s\n" "$response" | _egrep_o "\"id\":[^,]*" | cut -d : -f 2 | head -n 1)
+    record_id=$(printf "%s\n" "$response" | _egrep_o ",\"value\":\"..$txtvalue..\",\"id\":[^,]*" | cut -d : -f 3 | head -n 1)
     _debug "record_id" "$record_id"
     if [ -z "$record_id" ]; then
       _err "Can not get record id to remove."
@@ -152,7 +134,7 @@ _me_rest() {
   data="$3"
   _debug "$ep"
 
-  cdate=$(date -u +"%a, %d %b %Y %T %Z")
+  cdate=$(LANG=C date -u +"%a, %d %b %Y %T %Z")
   hmac=$(printf "%s" "$cdate" | _hmac sha1 "$(printf "%s" "$ME_Secret" | _hex_dump | tr -d " ")" hex)
 
   export _H1="x-dnsme-apiKey: $ME_Key"