Scratch Org

So it’s time to learn about Scratch Orgs. I know about them, but don’t know HOW to do them.

I have all my VSCode environment set up and I have a Dev Hub. My goal is to get the Dreamhouse App installed on a pre-release org (a few days after the release to Sandboxes).

I google getting started with scratch orgs and find this video https://www.youtube.com/watch?v=RJWZD2qO0Cw. Nope, this does not start at the beginning. It jumps straight to having a config file pre-set. Where do I get a config file? where do I save the config file? when do I set up my git repo?

Next Stop Trailhead https://trailhead.salesforce.com/content/learn/projects/quick-start-salesforce-dx/set-up-your-salesforce-dx-environment

OK, this is getting somewhere.

Set up the Scratch Org

First step log into the dev hub

sfdx force:auth:web:login -d -a DevHub

I’m following the trailhead because I want the Dreamhouse app, so that starts with a Git Clone so doesn’t help me set up a good process for setting up git in the first place. But it looks like Git Clone does make the folder git enabled, which is good.

Next step is that I connected my local Git folder to my Bitbucket via SourceTree, and checked out a branch to work on. (See my VS Code for Non Developers page to get help on how to do this part). So now we can start working. (Until I have everything set up, and backed up to the cloud, I don’t like to start).

I need to make one change to my project-scratch-def.json file to make it a preview release developer org.

"release": "Preview",

I run the command to create the Scratch Org, with the alias of JMDemo

sfdx force:org:create -s -f config/project-scratch-def.json -a JMDemo

So I start again without the Preview Org and run the command and I have a scratch org.

Open the scratch org with

sfdx force:org:open

And push the metadata in my folders in my local machine via

sfdx force:source:push

Then assign the perm set and import the data as per the trailhead module so I can get Dreamhouse app up and running.

sfdx force:org:open

And there it is, Dreamhouse app ready to go.

Now, Commit to ensure Git is all nice and clean ready for the next step.

Working on the Scratch Org

As I continue working, each thing I do in the org I use the command

sfdx force:source:pull

Which brings the metadata down to the source on my local machine. It’s good to do this after every main action you perform so you know that any changes you are making in the org are going to be in Source and therefore in a new Scratch Org. Eg I changed my User Name and pulled source but it did not pull the user details, of course, because that is data.

Issues

I had an issue where the pull command gave me the following error.

Entity of type 'CustomApplication' named 'Account' cannot be found

Weird

But this Stack Exchange Post kind of helped, I had to put Account.app in my .forceignore file and then it worked. Weird.

End of the Day

Remember to pull all your source from your org then commit your git and send it to your Bitbucket in the cloud.

Next Day

When opening VS Code again and trying to log back into the scratch org it was interesting that it wasn’t just connected. All the sandboxes I am connected to are still connected when I do force:org:list. So I had to do

sfdx force:org:open -u myusername

But nope, it was not even that. I had not closed VS Code down, I just re-opened it, but even doing a force:source:pull it says I was not in a SFDX project. Yet my path on my terminal was set exactly to the project I was happily working in yesterday. It’s these little things that annoy just trying to get everything up and running to work. OK, this stack exchange post seemed to get me back on the right track… so it looks like it’s an issue with starting from the Trailhead project. I just changed the directory to the dreamhouse-sfdx directory and it worked from there. It’s Weird because I was not in that directory yesterday.

Data Export and Import

Now we need to export some data… to save it to our Source… now of course I have gone and made it more complex and gone beyond the bounds of the stupidly simple examples that Salesforce give on their blog.

My data structure is now:

• Account has one or more Properties

• Properties may belong to one or more Accounts

• Broker has one or more Properties

• Properties are sold by one and only one Broker

So, Properties has two parent objects - Account and Broker. (Look, why Broker is not an Account or Contact in the Dreamhouse App, I don’t know - there is reasons why they would be either in the real world, but probably Account with one Contact if they are an Independent Broker).

The Data Tree Export only handles one relationship deep - eg you start with brokers and you get properties.

You can hack your Plan file as per this stack exchange post or this blog post but dammit, that blog post states you will have an issue with Record Types, and of course I have Record Types.

And data tree only handles 200 records (or you can use Shane Mc’s data tree which handles more records).

Some other options are to use regular import and export commands and ensure all your objects have ExternalIDs, or use a tool like ETCopyData or JSONBourne (for the really complex orgs).

For me for now, I’m just going to forget brokers, and manually deal with Record Types. It’s a cop out, I know, but this project is just to get me started with Scratch Orgs.

To understand how the data import works it’s a good idea to understand how the Composite APIs work. as the reference values are similar.

Remember, each time you add a new field or need some extra data in your scratch org, you need to amend your SOQL statements and update you tree data.

Install an App

Of course you will need to install apps in your Scratch Org. I need to install Doug Ayers Mass Action Scheduler. Thankfully Doug provided the Package IDs on his release page https://github.com/douglascayers-org/sfdx-mass-action-scheduler/wiki/Release-Notes but you can usually see them in the install URLs for the version you are installing.

sfdx force:package:install --package "04XXXXXXXXXXXX"

OMG how easy is that! (Of course I don’t know how easy it is to do with all the options you need like restricting profiles, but for regular installs it’s easy).

Doug’s app is one that requires a Connected App and Named Credential set up. So this is going to get difficult when setting up the scratch org again. The Named Credential comes down when I pull the source, but the endpoint is hard coded to the My Domain, which won’t exist in a new scratch org. So the Named Credential will have to go into .forceignore and these steps will have to be done manually. (There could be a way to do this with scripting but I don’t need to know this yet).

Also, the configuration in MAS is data and there is not an easy way to move configs between orgs yet, and it relies on the Named Credential so I will just have to create it each time… there is only one MAS configuration for my demo so that’s fine.

Get a Password

The last thing I need to do for my demo is export data from a report into Excel using Enabler4Excel. For that I need to be able to log into the org using OAuth, and therefore I need a Password. To get a password you do

sfdx force:user:password:generate

Interestingly I had to use Sandbox as my OAuth endpoint to login. I thought Developer Orgs were “Production” orgs - maybe scratch orgs are different or maybe because I am using my production org as the Dev Hub. Dunno.

Re-Creating the Scratch Org

So the whole point of the Scratch Org is to re-create it from Source.

Now that the preview error is sorted, I have a preview developer org.

So I do

sfdx force:source:push

Debugging Issues

And I get the dreaded message <unknown> duplicates value on record with id: <unknown>. This is where the time sink starts. How does a company the size of Salesforce allow an error of that level of bullshit even get out. It’s even worse than a Gack!

Plus I’ve got some flow errors that are related to new flow features that I’m not even using. (Similar to this).

Trying to work this out:

• Disable field tracking on the one object that has it

• Put my flows into .forceignore (that didn’t work).

• Disable field tracking on the two fields that have it.

• Delete flows completely (they are still in my git).

• Hmm, that worked. Weird.

• Revert my flows

• Nope, same error.

So there is an error with Flows. I don’t want to have to re-create my whole 4 flows again, since flows are the basis of my demo!

But something in that flow error article made me think of the API version. Since I’m going into a preview org, maybe I need to update the API. But it should not be the case, because it should be backwards compatible. This post tells me how to set the API version.

sfdx force:config:set apiVersion=48.0

But that did not work.

Ah, interesting, I think it IS something to do with my flows, my API version and the fact that I created a scratch org from a github repo. The repo is on version 45 based on metadataTypeInfos.json. But the org I built the Flows in is Winter '20 so that is version 47. And I did build a flow using a new feature with the Start Node as per the post mentioned above.

• So I delete that one flow.

• Nope.

• Delete all but the main flow.

• Nope.

• But I’m starting to get somewhere now.

Flow Issues

So it seems that the new features in Winter '20 in Flows, where you do not have to manually create all the variables (great feature) is having an issue with the metadata. This Post states that it was confirmed by support. I can not see it in Known Issues though. But it seems to be the cause of the duplicate values issue also. See I had 3 Get Records lookups, all getting the same field value and because it could not understand the metadata of the {!Property from Get_VicRates.Levy_Rate__c} syntax it could not determine that {!Property from Get_NSWRates.Levy_Rate__c} was a completely different field.

Now, the thing is, I am trying to demo Spring ‘20 features, that rely on these Winter '20 features! I don’t want old fashioned Flows in my new Spring '20 flows demo. I now have to rebuild my flows to be compatible with Summer '19 so I can deploy them to a scratch org. Annoying.

Just for shits and giggles I created a Winter '20 Scratch org and tried to push my flow, built in a Winter '20 scratch org, and pulled from a Winter '20 scratch org and it is exactly the same issue. It is nothing to do withe the Preview org. It seems that flow metadata support in DX is a version behind. (Yes, I DID update the CLI before I started working today). The error is…

Get_Property (Get Records) - Because the queriedFields field is set, you also need to set the following fields: sObjectOutputReference.

(There is probably something more deeply involved than this, and probably a way I could do something if I sat here for another 6 hours and googled, but it’s a way forward for now, so I have to take it).

Weird Things

The end goal for me would be to create a scratch org that I am happy using that has all my standard modifications in it (eg as shown in New Salesforce Org).

The default scratch org has way too many features in it and weird settings.

Features not needed (could be part of developer org standard)

• FSL

• Analytics

• Site dot com

• Work dot com

• Community

  • Lightning Bolt

• CPQ

• Orders

• Too many Profiles

Weird Settings

• Not Enhanced Profile View, but it does have Enhanced Profile List Views

Scratch Org Missing Features

What Scratch Orgs can’t do, and I would have to set up each time I re-generated the scratch org

• Tab Positions (but I will change the App instead).

• Profile settings of which Record Types are available or Default

• Enhanced Profile View

Good Things

• Developer Scratch orgs come with My Domain already set up.