So it’s time to learn about Scratch Orgs. I know about them, but don’t know HOW to do them.

I have all my VSCode environment set up and I have a Dev Hub. My goal is to get the Dreamhouse App installed on a pre-release org (a few days after the release to Sandboxes).

I google getting started with scratch orgs and find this video https://www.youtube.com/watch?v=RJWZD2qO0Cw. Nope, this does not start at the beginning. It jumps straight to having a config file pre-set. Where do I get a config file? where do I save the config file? when do I set up my git repo?

Next Stop Trailhead https://trailhead.salesforce.com/content/learn/projects/quick-start-salesforce-dx/set-up-your-salesforce-dx-environment

OK, this is getting somewhere.

Set up the Scratch Org

First step log into the dev hub

sfdx force:auth:web:login -d -a DevHub

I’m following the trailhead because I want the Dreamhouse app, so that starts with a Git Clone so doesn’t help me set up a good process for setting up git in the first place. But it looks like Git Clone does make the folder git enabled, which is good.

Next step is that I connected my local Git folder to my Bitbucket via SourceTree, and checked out a branch to work on. (See my VS Code for Non Developers page to get help on how to do this part). So now we can start working. (Until I have everything set up, and backed up to the cloud, I don’t like to start).

I need to make one change to my project-scratch-def.json file to make it a preview release developer org.

"release": "Preview",

I run the command to create the Scratch Org, with the alias of JMDemo

sfdx force:org:create -s -f config/project-scratch-def.json -a JMDemo

So I start again without the Preview Org and run the command and I have a scratch org.

Open the scratch org with

sfdx force:org:open

And push the metadata in my folders in my local machine via

sfdx force:source:push

Then assign the perm set and import the data as per the trailhead module so I can get Dreamhouse app up and running.

sfdx force:org:open

And there it is, Dreamhouse app ready to go.

Now, Commit to ensure Git is all nice and clean ready for the next step.

Working on the Scratch Org

As I continue working, each thing I do in the org I use the command

sfdx force:source:pull
or 
sfdx force:source:pull -u yourornameg@org.name -f 

Which brings the metadata down to the source on my local machine. It’s good to do this after every main action you perform so you know that any changes you are making in the org are going to be in Source and therefore in a new Scratch Org. Eg I changed my User Name and pulled source but it did not pull the user details, of course, because that is data.

-f is handy when you need to overwrite all the source in your local machine with what is on the org.

Issues

I had an issue where the pull command gave me the following error.

Entity of type 'CustomApplication' named 'Account' cannot be found

Weird

But this Stack Exchange Post kind of helped, I had to put Account.app in my .forceignore file and then it worked. Weird.

End of the Day

Remember to pull all your source from your org then commit your git and send it to your Bitbucket in the cloud.

I was only doing work in my org and doing pull. But I ended up getting quite stuck (see below), so I think it’s a good idea to push and pull often (but then I don’t know how you handle the things that you can’t store in Scratch Orgs easily, like Connected Apps and things that have a hard coded reference to the Org Name, but that is for another time).

Next Day

When opening VS Code again and trying to log back into the scratch org it was interesting that it wasn’t just connected. All the sandboxes I am connected to are still connected when I do force:org:list. So I had to do

sfdx force:org:open -u myusername

But nope, it was not even that. I had not closed VS Code down, I just re-opened it, but even doing a force:source:pull it says I was not in a SFDX project. Yet my path on my terminal was set exactly to the project I was happily working in yesterday. It’s these little things that annoy just trying to get everything up and running to work. OK, this stack exchange post seemed to get me back on the right track… so it looks like it’s an issue with starting from the Trailhead project. I just changed the directory to the dreamhouse-sfdx directory and it worked from there. It’s Weird because I was not in that directory yesterday.

Data Export and Import

Now we need to export some data… to save it to our Source… now of course I have gone and made it more complex and gone beyond the bounds of the stupidly simple examples that Salesforce give on their blog.

My data structure is now:

• Account has one or more Properties

• Properties may belong to one or more Accounts

• Broker has one or more Properties

• Properties are sold by one and only one Broker

So, Properties has two parent objects - Account and Broker. (Look, why Broker is not an Account or Contact in the Dreamhouse App, I don’t know - there is reasons why they would be either in the real world, but probably Account with one Contact if they are an Independent Broker).

The Data Tree Export only handles one relationship deep - eg you start with brokers and you get properties.

You can hack your Plan file as per this stack exchange post or this blog post but dammit, that blog post states you will have an issue with Record Types, and of course I have Record Types.

And data tree only handles 200 records (or you can use Shane Mc’s data tree which handles more records).

Some other options are to use regular import and export commands and ensure all your objects have ExternalIDs, or use a tool like ETCopyData or JSONBourne (for the really complex orgs).

To understand how the data import works it’s a good idea to understand how the Composite APIs work. as the reference values are similar.

Remember, each time you add a new field or need some extra data in your scratch org, you need to amend your SOQL statements and update you tree data.

Issues - Location data

Ah here’s a gotcha. Any location fields can’t be exported directly with force:tree:export - well they can be, but then they will error on force:tree:import saying Location__c is not a valid field.

You have to export with this syntax

"SELECT Name, Location__Latitude__s, Location__Longitude__s FROM Account"

Giving Up on Data

OK, for my demo I’m going to give up using dx commands to deal with data. It’s a cop out, I know, but this project is just to get me started with Scratch Orgs.

Install an App

Of course you will need to install apps in your Scratch Org. I need to install Doug Ayers Mass Action Scheduler. Thankfully Doug provided the Package IDs on his release page https://github.com/douglascayers-org/sfdx-mass-action-scheduler/wiki/Release-Notes but you can usually see them in the install URLs for the version you are installing.

sfdx force:package:install --package "04XXXXXXXXXXXX"

OMG how easy is that! (Of course I don’t know how easy it is to do with all the options you need like restricting profiles, but for regular installs it’s easy).

Doug’s app is one that requires a Connected App and Named Credential set up. So this is going to get difficult when setting up the scratch org again. The Named Credential comes down when I pull the source, but the endpoint is hard coded to the My Domain, which won’t exist in a new scratch org. So the Named Credential will have to go into .forceignore and these steps will have to be done manually. (There could be a way to do this with scripting but I don’t need to know this yet).

Also, the configuration in MAS is data and there is not an easy way to move configs between orgs yet, and it relies on the Named Credential so I will just have to create it each time… there is only one MAS configuration for my demo so that’s fine.

Get a Password

The last thing I need to do for my demo is export data from a report into Excel using Enabler4Excel. For that I need to be able to log into the org using OAuth, and therefore I need a Password. To get a password you do

sfdx force:user:password:generate

Interestingly I had to use Sandbox as my OAuth endpoint to login. I thought Developer Orgs were “Production” orgs - maybe scratch orgs are different or maybe because I am using my production org as the Dev Hub. Dunno.

Re-Creating the Scratch Org

So the whole point of the Scratch Org is to re-create it from Source.

Now that the preview error is sorted, I have a preview developer org.

So I do

sfdx force:source:push

Errors

Source Errors

And then things from there went to hell for me. 4+ hours of trying to work this out, and in the end René Winkelmeyer asked “what's the error that you see in the org?”. Aha! something that I should have known and should have been obvious, but gee I felt like such and idiot that I got so bogged down in googling the errors that I did not step back and reason how metadata gets into an org.

Now firstly the error is VS Code is stupid and ridiculous and should be way better.

N/A duplicate value found: <unknown> duplicates value on record with id: <unknown>

That’s it! That’s the error. Googling, this article <unknown> duplicates value on record with id: <unknown> is useful but it started me on a huge time sink. How does a company the size of Salesforce allow an error of that level of bullshit even get out. It’s even worse than a Gack! At least a Gack has a number and you can ask some kind soul who works at Salesforce to look it up. But this error, and the Stack Exchange post and every single other post I looked did not give me the help I needed. When I found the help I needed - exactly what was causing this error it was easy to spot and fix.

Do not waste 4 hours like I did! (but yes, there were other errors with flow, see below).

So for me, the error was that I had created a report in public reports, pulled the metadata, then moved the report into an existing folder then pulled the metadata at another time. There is something funky with report folder metadata and it took a workaround and some help from James Melville to delete the offending source, and then push to my new org, and remove the duplicate issue. But then in the midst of that there was also flow errors.

The other thing René asked me is had I tried doing a regular deploy and retrieve. I had tried to google using push to push one piece of metadata like I can with DX for Deployments, but a quick google told me that was not possible. So once again, I let googling get the better of me rather than my knowledge of how metadata deployments work. I got caught up in the belief that scratch orgs are somehow completely different and did not think that I could just use force:source:deploy to deploy one item of metadata.

After the 4+ hours, my work around was definitely going to be to simplify my metadata and deploy to a developer org or a pre-release org and not use the Dreamhouse basis for my demo. I think starting with that has caused many of my issues.

Flow Errors

I’ve got some flow errors that are related to new flow features that I’m not even using. (Similar to this).

I don’t want to have to re-create my whole 4 flows again, since flows are the basis of my demo!

But something in that flow error article made me think of the API version. Since I’m going into a preview org, maybe I need to update the API. But it should not be the case, because it should be backwards compatible. This post tells me how to set the API version.

sfdx force:config:set apiVersion=48.0

But that did not work.

Ah, interesting, I think it IS something to do with my flows, my API version and the fact that I created a scratch org from a github repo. The repo is on version 45 based on metadataTypeInfos.json. But the org I built the Flows in is Winter '20 so that is version 47. And I did build a flow using a new feature with the Start Node as per the post mentioned above.

• So I delete that one flow.

• Nope.

• Delete all but the main flow.

• Nope.

• But I’m starting to get somewhere now.

So it seems that the new features in Winter '20 in Flows, where you do not have to manually create all the variables (great feature) is having an issue with the metadata. This Post states that it was confirmed by support. I can not see it in Known Issues though.

Now, the thing is, I am trying to demo Spring ‘20 features, that rely on these Winter '20 features! I don’t want old fashioned Flows in my new Spring '20 flows demo. I now have to rebuild my flows to be compatible with Summer '19 so I can deploy them to a scratch org. Annoying.

Just for shits and giggles I created a Winter '20 Scratch org and tried to push my flow, built in a Winter '20 scratch org, and pulled from a Winter '20 scratch org and it is exactly the same issue. It is nothing to do withe the Preview org. It seems that flow metadata support in DX is a version behind. (Yes, I DID update the CLI before I started working today). The error is…

Get_Property (Get Records) - Because the queriedFields field is set, you also need to set the following fields: sObjectOutputReference.

(There is probably something more deeply involved than this, and probably a way I could do something if I sat here for another 6 hours and googled, but there is a way forward for now, so I have to take it. I had to remove the Winter '20 features from my flows).

API Issues

So I found that my sfdx-project.json was set at version 45 - what the version was when the dreamhouse org was created. I tried to set the version to 47 more errors! (eg enableChangeDataCapture was deprecated in 46).

So, I’m trying to push a version 47 feature into an org whose metadata source is on version 45 and the only way around that is to hack the metadata to comply with the new versions. Gee this sucks. (I know why it is so, but wow, to have to do all this with each release on a huge org where you are trying to use a fully automated CI pipeline, kudos to you, because this sucks).

So I cleaned up the deprecated things, and still nope. I still think there is an issue with the flow specific metadata for 47.

Delete a Scratch Org

sfdx force:org:delete -u yourorgname@org.name

Expired Scratch Org

So after ALL my issues, my first scratch org where I had my full running demo is now on it’s last day. Dammit. Once again, I’m not delving into ALL the learning about how to do Named Credentials and data imports etc, so I don’t have a fully featured org in my source.

So I went low-tech and videoed all the changes that I did in my org so I can rebuild it in my new scratch org as needed.

Scripting

Now, after doing all of this I’ve learned a heap, and had sooo many frustrations that I don’t think I will delve into scratch orgs again in a hurry. They are definitely not a panacea. I like my sandbox environments where I can control what is in the org, and have my org as my source of truth (There! I said it! I’m obviously not a real developer. Deal with it!), and have my git as my backup.

But the first thing I would do now is keep a file of all the commands I need to use, and the order they are needed in. It would be nice to have a fully scripted CI/CD process but I’m not going to go down that path, but there is definitely value in having the list of commands you need to use for your project there and ready to copy and paste on the command line.

Yep, and I’m not learning Bash any time soon (Again! Not a real developer!), a smattering of Powershell maybe.

Scratch Org Notes

Weird Things

The end goal for me would be to create a scratch org that I am happy using that has all my standard modifications in it (eg as shown in New Salesforce Org).

The default scratch org has way too many features in it and weird settings.

Features not needed (could be part of developer org standard)

• FSL

• Analytics

• Site dot com

• Work dot com

• Community

  • Lightning Bolt

• CPQ

• Orders

• Too many Profiles

Weird Settings

• Not Enhanced Profile View, but it does have Enhanced Profile List Views

Scratch Org Missing Features

What Scratch Orgs can’t do, and I would have to set up each time I re-generated the scratch org

• Tab Positions (but I will change the App instead).

• Profile settings of which Record Types are available or Default

• Enhanced Profile View

Good Things

• Developer Scratch orgs come with My Domain already set up.