Page Comparison

...

Remember to pull all your source from your org then commit your git and send it to your Bitbucket in the cloud.

Next Day

When opening VS Code again and trying to log back into the scratch org it was interesting that it wasn’t just connected. All the sandboxes I am connected to are still connected when I do force:org:list. So I had to do

sfdx force:org:open -u myusername

But nope, it was not even that. I had not closed VS Code down, I just re-opened it, but even doing a force:source:pull it says I was not in a SFDX project. Yet my path on my terminal was set exactly to the project I was happily working in yesterday. It’s these little things that annoy just trying to get everything up and running to work. OK, this stack exchange post seemed to get me back on the right track… so it looks like it’s an issue with starting from the Trailhead project. I just changed the directory to the dreamhouse-sfdx directory and it worked from there. It’s Weird because I was not in that directory yesterday.

Data Export and Import

Now we need to export some data… to save it to our Source… now of course I have gone and made it more complex and gone beyond the bounds of the stupidly simple examples that Salesforce give on their blog.

My data structure is now:

• Account has one or more Properties

• Properties may belong to one or more Accounts

• Broker has one or more Properties

• Properties are sold by one and only one Broker

So, Properties has two parent objects - Account and Broker. (Look, why Broker is not an Account or Contact in the Dreamhouse App, I don’t know - there is reasons why they would be either in the real world, but probably Account with one Contact if they are an Independent Broker).

The Data Tree Export only handles one relationship deep - eg you start with brokers and you get properties.

You can hack your Plan file as per this stack exchange post or this blog post but dammit, that blog post states you will have an issue with Record Types, and of course I have Record Types.

And data tree only handles 200 records (or you can use Shane Mc’s data tree which handles more records).

Some other options are to use regular import and export commands and ensure all your objects have ExternalIDs, or use a tool like ETCopyData or JSONBourne (for the really complex orgs).

For me for now, I’m just going to forget brokers, and manually deal with Record Types. It’s a cop out, I know, but this project is just to get me started with Scratch Orgs.

To understand how the data import works it’s a good idea to understand how the Composite APIs work. as the reference values are similar.

Remember, each time you add a new field or need some extra data in your scratch org, you need to amend your SOQL statements and update you tree data.

Install an App

Of course you will need to install apps in your Scratch Org. I need to install Doug Ayers Mass Action Scheduler. Thankfully Doug provided the Package IDs on his release page https://github.com/douglascayers-org/sfdx-mass-action-scheduler/wiki/Release-Notes but you can usually see them in the install URLs for the version you are installing.

sfdx force:package:install --package "04XXXXXXXXXXXX"

OMG how easy is that! (Of course I don’t know how easy it is to do with all the options you need like restricting profiles, but for regular installs it’s easy).

Doug’s app is one that requires a Connected App and Named Credential set up. So this is going to get difficult when setting up the scratch org again. The Named Credential comes down when I pull the source, but the endpoint is hard coded to the My Domain, which won’t exist in a new scratch org. So the Named Credential will have to go into .forceignore and these steps will have to be done manually. (There could be a way to do this with scripting but I don’t need to know this yet).

Also, the configuration in MAS is data and there is not an easy way to move configs between orgs yet, and it relies on the Named Credential so I will just have to create it each time… there is only one MAS configuration for my demo so that’s fine.

Get a Password

The last thing I need to do for my demo is export data from a report into Excel using Enabler4Excel. For that I need to be able to log into the org using OAuth, and therefore I need a Password. To get a password you do

sfdx force:user:password:generate

Interestingly I had to use Sandbox as my OAuth endpoint to login. I thought Developer Orgs were “Production” orgs - maybe scratch orgs are different or maybe because I am using my production org as the Dev Hub. Dunno.

Re-Creating the Scratch Org

So the whole point of the Scratch Org is to re-create it from Source.

Now that the preview error is sorted, I have a preview developer org.

So I do

sfdx force:source:push

Debugging Issues

And I get the dreaded message <unknown> duplicates value on record with id: <unknown>. This is where the time sink starts. How does a company the size of Salesforce allow an error of that level of bullshit even get out. It’s even worse than a Gack!

Plus I’ve got some flow errors that are related to new flow features that I’m not even using. (Similar to this).

Trying to work this out:

• Disable field tracking on the one object that has it

• Put my flows into .forceignore (that didn’t work).

• Disable field tracking on the two fields that have it.

• Delete flows completely (they are still in my git).

• Hmm, that worked. Weird.

• Revert my flows

• Nope, same error.

So there is an error with Flows. I was only doing work in my org and doing pull. But I ended up getting quite stuck (see below), so I think it’s a good idea to push and pull often (but then I don’t know how you handle the things that you can’t store in Scratch Orgs easily, like Connected Apps and things that have a hard coded reference to the Org Name, but that is for another time).

Next Day

When opening VS Code again and trying to log back into the scratch org it was interesting that it wasn’t just connected. All the sandboxes I am connected to are still connected when I do force:org:list. So I had to do

sfdx force:org:open -u myusername

But nope, it was not even that. I had not closed VS Code down, I just re-opened it, but even doing a force:source:pull it says I was not in a SFDX project. Yet my path on my terminal was set exactly to the project I was happily working in yesterday. It’s these little things that annoy just trying to get everything up and running to work. OK, this stack exchange post seemed to get me back on the right track… so it looks like it’s an issue with starting from the Trailhead project. I just changed the directory to the dreamhouse-sfdx directory and it worked from there. It’s Weird because I was not in that directory yesterday.

Data Export and Import

Now we need to export some data… to save it to our Source… now of course I have gone and made it more complex and gone beyond the bounds of the stupidly simple examples that Salesforce give on their blog.

My data structure is now:

• Account has one or more Properties

• Properties may belong to one or more Accounts

• Broker has one or more Properties

• Properties are sold by one and only one Broker

So, Properties has two parent objects - Account and Broker. (Look, why Broker is not an Account or Contact in the Dreamhouse App, I don’t know - there is reasons why they would be either in the real world, but probably Account with one Contact if they are an Independent Broker).

The Data Tree Export only handles one relationship deep - eg you start with brokers and you get properties.

You can hack your Plan file as per this stack exchange post or this blog post but dammit, that blog post states you will have an issue with Record Types, and of course I have Record Types.

And data tree only handles 200 records (or you can use Shane Mc’s data tree which handles more records).

Some other options are to use regular import and export commands and ensure all your objects have ExternalIDs, or use a tool like ETCopyData or JSONBourne (for the really complex orgs).

For me for now, I’m just going to forget brokers, and manually deal with Record Types. It’s a cop out, I know, but this project is just to get me started with Scratch Orgs.

To understand how the data import works it’s a good idea to understand how the Composite APIs work. as the reference values are similar.

Remember, each time you add a new field or need some extra data in your scratch org, you need to amend your SOQL statements and update you tree data.

Install an App

Of course you will need to install apps in your Scratch Org. I need to install Doug Ayers Mass Action Scheduler. Thankfully Doug provided the Package IDs on his release page https://github.com/douglascayers-org/sfdx-mass-action-scheduler/wiki/Release-Notes but you can usually see them in the install URLs for the version you are installing.

sfdx force:package:install --package "04XXXXXXXXXXXX"

OMG how easy is that! (Of course I don’t know how easy it is to do with all the options you need like restricting profiles, but for regular installs it’s easy).

Doug’s app is one that requires a Connected App and Named Credential set up. So this is going to get difficult when setting up the scratch org again. The Named Credential comes down when I pull the source, but the endpoint is hard coded to the My Domain, which won’t exist in a new scratch org. So the Named Credential will have to go into .forceignore and these steps will have to be done manually. (There could be a way to do this with scripting but I don’t need to know this yet).

Also, the configuration in MAS is data and there is not an easy way to move configs between orgs yet, and it relies on the Named Credential so I will just have to create it each time… there is only one MAS configuration for my demo so that’s fine.

Get a Password

The last thing I need to do for my demo is export data from a report into Excel using Enabler4Excel. For that I need to be able to log into the org using OAuth, and therefore I need a Password. To get a password you do

sfdx force:user:password:generate

Interestingly I had to use Sandbox as my OAuth endpoint to login. I thought Developer Orgs were “Production” orgs - maybe scratch orgs are different or maybe because I am using my production org as the Dev Hub. Dunno.

Re-Creating the Scratch Org

So the whole point of the Scratch Org is to re-create it from Source.

Now that the preview error is sorted, I have a preview developer org.

So I do

sfdx force:source:push

Errors

Source Errors

And then things from there went to hell for me. 4+ hours of trying to work this out, and in the end René Winkelmeyer asked “what's the error that you see in the org?”. Aha! something that I should have known and should have been obvious, but gee I felt like such and idiot that I got so bogged down in googling the errors that I did not step back and reason how metadata gets into an org.

Now firstly the error is VS Code is stupid and ridiculous and should be way better.

N/A duplicate value found: <unknown> duplicates value on record with id: <unknown>

That’s it! That’s the error. Googling, this article <unknown> duplicates value on record with id: <unknown> is useful but it started me on a huge time sink. How does a company the size of Salesforce allow an error of that level of bullshit even get out. It’s even worse than a Gack! At least a Gack has a number and you can ask some kind soul who works at Salesforce to look it up. But this error, and the Stack Exchange post and every single other post I looked did not give me the help I needed. When I found the help I needed - exactly what was causing this error it was easy to spot and fix.

Do not waste 4 hours like I did! (but yes, there were other errors with flow, see below).

So for me, the error was that I had created a report in public reports, pulled the metadata, then moved the report into an existing folder then pulled the metadata at another time. There is something funky with report folder metadata and it took a workaround and some help from James Melville to delete the offending source, and then push to my new org, and remove the duplicate issue. But then in the midst of that there was also flow errors.

The other thing René asked me is had I tried doing a regular deploy and retrieve. I had tried to google using push to push one piece of metadata like I can with DX for Deployments, but a quick google told me that was not possible. So once again, I let googling get the better of me rather than my knowledge of how metadata deployments work. I got caught up in the belief that scratch orgs are somehow completely different and did not think that I could just use force:source:deploy to deploy one item of metadata.

After the 4+ hours, my work around was definitely going to be to simplify my metadata and deploy to a developer org or a pre-release org and not use the Dreamhouse basis for my demo. I think starting with that has caused many of my issues.

Flow Errors

I’ve got some flow errors that are related to new flow features that I’m not even using. (Similar to this).

I don’t want to have to re-create my whole 4 flows again, since flows are the basis of my demo!

...

Ah, interesting, I think it IS something to do with my flows, my API version and the fact that I created a scratch org from a github repo. The repo is on version 45 based on metadataTypeInfos.json. But the org I built the Flows in is Winter '20 so that is version 47. And I did build a flow using a new feature with the Start Node as per the post mentioned above.

...

So I delete that one flow.

...

Nope.

...

Delete all but the main flow.

...

Nope.

...

.

...

• So

...

• I delete that one flow.

• Nope.

• Delete all but the main flow.

• Nope.

• But I’m starting to get somewhere now.

So it seems that the new features in Winter '20 in Flows, where you do not have to manually create all the variables (great feature) is having an issue with the metadata. This Post states that it was confirmed by support. I can not see it in Known Issues though.

Now, the thing is, I am trying to demo Spring ‘20 features, that rely on these Winter '20 features! I don’t want old fashioned Flows in my new Spring '20 flows demo. I now have to rebuild my flows to be compatible with Summer '19 so I can deploy them to a scratch org. Annoying.

...

(There is probably something more deeply involved than this, and probably a way I could do something if I sat here for another 6 hours and googled, but it’s there is a way forward for now, so I have to take it. I had to remove the Winter '20 features from my flows).

API Issues

Ah, it is waaaay more than that! Wow. So I found that my sfdx-project.json was set at version 45 - what the version was when the dreamhouse org was created. I tried to set the version to 47 and all hell broke loose! Way to many errors. more errors! (eg enableChangeDataCapture was deprecated in 46).

So, I’m trying to push a version 47 feature into an org whose metadata source is on version 45 and the only way around that is to hack the metadata to comply with the new versions. Gee this sucks. (I know why it is so, but wow, to have to do all this with each release on a huge org where you are trying to use a fully automated CI pipeline, kudos to you, because this sucks).

...

sfdx force:org:delete -u yourorgname@org.name

Expired Scratch Org

So after ALL my issues, my first scratch org where I had my full running demo is now expired. Dammit. Once again, I’m not delving into ALL the learning about how to do Named Credentials etc, so I don’t have a fully featured org in my source.

Weird Things

The end goal for me would be to create a scratch org that I am happy using that has all my standard modifications in it (eg as shown in New Salesforce Org).

...