need to transfer infrastructure to OpenStreetMap US #111

d3netxer · 2019-10-21T18:26:31Z

Description

The goal is to transfer the backend infrastructure to OpenStreetMap US.

d3netxer · 2019-10-21T18:30:32Z

Requesting to OpenStreetMap US (@iandees) to implement the following:

Create the following two buckets in US-East:

teachosm-project-pics-deployed
teachosm-project-content-original-deployed

These two buckets will have ‘Block all public access’ turned off, in the Access control list for the ‘everyone’ group all boxes will be unchecked, and the following will be entered and saved for the CORS configuration:

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
    <AllowedOrigin>*</AllowedOrigin>
    <AllowedMethod>GET</AllowedMethod>
    <AllowedMethod>PUT</AllowedMethod>
    <AllowedMethod>POST</AllowedMethod>
    <AllowedMethod>HEAD</AllowedMethod>
    <AllowedHeader>*</AllowedHeader>
</CORSRule>
</CORSConfiguration>

Also create the following two buckets in US-East:

teachosm-project-content-deployed
teachosm-project-posts-deployed
teachosm-libreoffice-deployed

These three buckets will have ‘Block all public access’ turned off, in the Access control list for the ‘everyone’ group all boxes will be unchecked, and the default CORS configuration can be left as is.

iandees · 2019-10-28T14:14:50Z

I created the following buckets:

teachosm-project-pics-deployed
teachosm-project-content-original-deployed
teachosm-project-content-deployed
teachosm-project-posts-deployed
teachosm-libreoffice-deployed

And disabled block all public access. The first two have CORS configurations added.

Remaining when we're farther along:

Adjust AllowedOrigin on the CORS configuration so that it only loads on OSM US-hosted sites.
Put domain and/or CDN in front of the S3 buckets and don't allow public access to the S3 buckets

d3netxer · 2019-10-29T21:23:11Z

I tested deploying today, but need some additional adjustments from @iandees :

use this advanced permissions template for reference (https://serverless-stack.com/chapters/customize-the-serverless-iam-policy.html) and for the cloudformation policies adjust the resource to this: arn:aws:cloudformation:us-east-1:509553493608:stack/imageUploadDemo-dev/*
create the following two new deployment buckets in us-east-1:

convert-post-request-to-yaml-deploy
serverlessS3Upload-deploy

These two buckets will have ‘Block all public access’ turned off, in the Access control list for the ‘everyone’ group all boxes will be unchecked, and the default CORS configuration can be left as is.

iandees · 2019-11-11T15:49:36Z

I added arn:aws:cloudformation:us-east-1:509553493608:stack/imageUploadDemo-dev/* to the list of cloudformation resources in the existing policy on your account.
Added S3 bucket convert-post-request-to-yaml-deploy. Bucket serverlessS3Upload-deploy is not possible because bucket names can only have lower case letters.

d3netxer · 2019-11-14T13:50:41Z

ok, for item 2, please create this bucket instead: teachosm-serverless-s3upload-deploy

Also, I am having trouble with the permissions for the convert-post-request-to-yaml-deploy bucket. The Serverless Error is saying: "Could not locate deployment bucket. Error: Access Denied".

I will get with you offline about this.

iandees · 2019-11-14T15:59:44Z

As a reminder, your account has access to do anything to buckets with the prefix teachosm-geosurge-. You should be able to create buckets, list them, etc. as long as they have that prefix.

In the meantime, I created the teachosm-serverless-s3upload-deploy bucket and added convert-post-request-to-yaml-deploy to the list of buckets you can write to.

d3netxer · 2019-11-14T17:01:54Z

Thanks, noted about access using the prefix.

I just keep on moving forward using the teachosm-serverless-s3upload-deploy and "convert-post-request-to-yaml-deploy" buckets.

I ran the deploy command again and I was allowed to move forward. However, a new error now appears that says my user is not allowed to perform apigateway:PATCH on resource.

I think this can be resolved with another edit to the permissions. In the existing permissions, there should be a block like this:

{
      "Effect": "Allow",
      "Action": [
        "apigateway:GET",
        "apigateway:POST",
        "apigateway:PUT",
        "apigateway:DELETE"
      ],
      "Resource": [
        "arn:aws:apigateway:<region>::/restapis"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "apigateway:GET",
        "apigateway:POST",
        "apigateway:PUT",
        "apigateway:DELETE"
      ],
      "Resource": [
        "arn:aws:apigateway:<region>::/restapis/*"
      ]
    },

Please make sure that the is 'us-east-1'. Also add the "apigateway:PATCH" to the Actions.

iandees · 2019-11-14T19:49:48Z

I added PATCH to the list of Actions.

d3netxer · 2019-11-14T21:23:10Z

Thanks, the convert-post-request-to-yaml lambda function deployed successfully.

I am now getting a new error for the serverlessS3Upload function related to permissions for the cloudformation:DescribeStacks. Can you make adjustments to the permissions to the block that looks like this:

{
      "Effect": "Allow",
      "Action": [
        "cloudformation:Describe*",
        "cloudformation:List*",
        "cloudformation:Get*",
        "cloudformation:CreateStack",
        "cloudformation:UpdateStack",
        "cloudformation:DeleteStack"
      ],
      "Resource": "arn:aws:cloudformation:<region>:<account_no>:stack/<service_name>*/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "cloudformation:ValidateTemplate"
      ],
      "Resource": "*"
    },

I'm assuming the correct region and account_no are already inserted here. For the service name, you can input 'image-upload-prod'.

iandees · 2019-11-15T01:14:10Z

image-upload-prod is now in the CloudFormation list.

d3netxer · 2019-11-15T17:15:03Z

Thanks, it got past this step now.

The next permissions error is related to an Error for IamRoleLambdaExecution - API: iam:GetRole on on resource: role image-upload-prod-us-east-1-lambdaRole. Can you modify this block of the permissions:

{
      "Effect": "Allow",
      "Action": [
        "iam:GetRole",
        "iam:PassRole",
        "iam:CreateRole",
        "iam:DeleteRole",
        "iam:DetachRolePolicy",
        "iam:PutRolePolicy",
        "iam:AttachRolePolicy",
        "iam:DeleteRolePolicy"
      ],
      "Resource": [
        "arn:aws:iam::<account_no>:role/<service_name>*-lambdaRole"
      ]
    },

and insert 'image-upload-prod' for the <service_name> ?

iandees · 2019-11-15T17:22:13Z

Done!

d3netxer · 2019-11-15T17:47:37Z

Excellent. This worked, so these 2 lambdas are deployed: serverlessS3Upload and convert-post-request-to-yaml. There is a final one libreoffice-convert-to-pdf that I need to deploy, but I don't have a serverless script for this. My plan is begin trying next week to deploy using AWS CLI.

d3netxer · 2019-11-17T03:06:18Z

I have started trying to deploy the final function, using Serverless. Similar to the previous function, I got a cloudformation:DescribeStacks error. Can you make an adjustment in this block:

{
      "Effect": "Allow",
      "Action": [
        "cloudformation:Describe*",
        "cloudformation:List*",
        "cloudformation:Get*",
        "cloudformation:CreateStack",
        "cloudformation:UpdateStack",
        "cloudformation:DeleteStack"
      ],
      "Resource": "arn:aws:cloudformation:<region>:<account_no>:stack/<service_name>*/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "cloudformation:ValidateTemplate"
      ],
      "Resource": "*"
    },

You can add another resource by adding a comma followed by a second item. In the second item can you add 'geosurge-libreoffice-convert-to-pdf*' for the <service_name> ?

Also, let's modify the AWS role permissions as well. Looking at this block:

{
      "Effect": "Allow",
      "Action": [
        "iam:GetRole",
        "iam:PassRole",
        "iam:CreateRole",
        "iam:DeleteRole",
        "iam:DetachRolePolicy",
        "iam:PutRolePolicy",
        "iam:AttachRolePolicy",
        "iam:DeleteRolePolicy"
      ],
      "Resource": [
        "arn:aws:iam::<account_no>:role/<service_name>*-lambdaRole"
      ]
    },

Can you add another resource item and again add 'geosurge-libreoffice-convert-to-pdf*' for the <service_name> ?

Thanks

iandees · 2019-11-17T03:22:19Z

Done!

d3netxer · 2019-11-17T03:56:45Z

I'm getting this error, maybe to do with formatting. This link may help: https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_resource.html

Error --------------------------------------------------
 
  The CloudFormation template is invalid: Template format error: At least one Resources member must be defined.

d3netxer · 2019-12-27T00:43:54Z

@iandees I spent sometime making improvements and testing a whole re-deployment of all the lambdas on another AWS account. I have a new IAM policy that I hope will take care of all the permission issues here: https://gist.github.com/d3netxer/b1d1a4012d6bf20b910c22d02ee43a80

Can you try replacing the policy with this one? You just need to replace the and <account_no> wherever needed. Thanks

iandees · 2019-12-28T20:00:12Z

Change is made. It looks like you've been deploying to us-east-1, so I replaced <region> with that, too.

d3netxer · 2019-12-31T14:56:46Z

Excellent. All of the functions deployed successfully. A few modifications are left to complete the deployment on the live site.

One of the Lambda functions makes a pull request to the Github repo. For this I believe we need to post as a user who has admin access to the repo, along with a personal access token. @iandees you can either create a new Github user or use an existing one. The pull requests will be posted as that user.

that user will need to have access to the repo: https://github.com/osmus/teachosm.org
create a personal access token for the user (Go to settings --> developer settings --> personal access tokens, and create one with the repo and admin: repo_hook scope boxes checked)
pass along the username and personal access token to me privately

iandees · 2020-01-10T03:19:50Z

This is done. The user is teachosm-bot. I shared the access token privately.

d3netxer · 2020-01-10T03:31:59Z

thanks. I am getting close, today I switched out the backends, including copying the files between s3 buckets. I need to look at the libreoffice converter a little bit more because it isn't converting pdfs to pdfs. It probably shouldn't do this, if the file doesn't need to be converted, it should just be copied from the contents upload folder to the contents folder instead; so adding an if statement should fix this issue most likely. Right now 2 project download links are broken because of this. I will try to continue fixing this and doing more tests within the next few days.

d3netxer added in progress high priority labels Oct 21, 2019

d3netxer assigned iandees and d3netxer Oct 21, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

need to transfer infrastructure to OpenStreetMap US #111

need to transfer infrastructure to OpenStreetMap US #111

d3netxer commented Oct 21, 2019

d3netxer commented Oct 21, 2019

iandees commented Oct 28, 2019

d3netxer commented Oct 29, 2019

iandees commented Nov 11, 2019

d3netxer commented Nov 14, 2019

iandees commented Nov 14, 2019 •

edited

Loading

d3netxer commented Nov 14, 2019

iandees commented Nov 14, 2019

d3netxer commented Nov 14, 2019

iandees commented Nov 15, 2019

d3netxer commented Nov 15, 2019

iandees commented Nov 15, 2019

d3netxer commented Nov 15, 2019

d3netxer commented Nov 17, 2019

iandees commented Nov 17, 2019

d3netxer commented Nov 17, 2019

d3netxer commented Dec 27, 2019

iandees commented Dec 28, 2019

d3netxer commented Dec 31, 2019

iandees commented Jan 10, 2020

d3netxer commented Jan 10, 2020

need to transfer infrastructure to OpenStreetMap US #111

need to transfer infrastructure to OpenStreetMap US #111

Comments

d3netxer commented Oct 21, 2019

Description

d3netxer commented Oct 21, 2019

iandees commented Oct 28, 2019

d3netxer commented Oct 29, 2019

iandees commented Nov 11, 2019

d3netxer commented Nov 14, 2019

iandees commented Nov 14, 2019 • edited Loading

d3netxer commented Nov 14, 2019

iandees commented Nov 14, 2019

d3netxer commented Nov 14, 2019

iandees commented Nov 15, 2019

d3netxer commented Nov 15, 2019

iandees commented Nov 15, 2019

d3netxer commented Nov 15, 2019

d3netxer commented Nov 17, 2019

iandees commented Nov 17, 2019

d3netxer commented Nov 17, 2019

d3netxer commented Dec 27, 2019

iandees commented Dec 28, 2019

d3netxer commented Dec 31, 2019

iandees commented Jan 10, 2020

d3netxer commented Jan 10, 2020

iandees commented Nov 14, 2019 •

edited

Loading