FYI @PiotrZierhoffer -- I could use some help here

I tried to run it locally, and it takes a very long time to finish. I believe it's a plain timeout on lack of output, but I don't know how to force travis to accept this

@xobs - This is one of the issues I could use Window users help with...

• https://conda.io/projects/conda/en/latest/user-guide/install/windows.html#install-win-silent (and https://conda.io/projects/conda/en/latest/user-guide/install/linux.html#install-linux-silent)

• https://docs.anaconda.com/anaconda/install/silent-mode/

• https://github.com/litex-hub/litex-conda-misc/blob/master/conda-get.sh#L10

Installing on Windows — conda 4.8.3.post59+b7bd3866 documentation

Installing on Linux — conda 4.8.3.post59+b7bd3866 documentation

Installing in silent mode — Anaconda documentation

GitHub

litex-hub/litex-conda-misc

Contribute to litex-hub/litex-conda-misc development by creating an account on GitHub.

I think the settings we want are for conda to be as self contained as possible. I think this means we want;

• /InstallationType=JustMe
• /AddToPath=0
• /RegisterPython=0 -- JustMe
• /NoRegistry=0
• /NoScripts=0

The issue seems to be some interaction between bash, Makefile and Windows commands.

There seems to be some issue with start?

tim@tim-win-test MINGW64 ~/conda-env-make/test (master)
$ start "" /WAIT env/downloads/Miniconda3-latest-Windows-x86_64.exe /S /D=C:\temp2
The system cannot find the file C:/Program Files/Git/WAIT.

if you are doing something like start at at $ prompt, I can only assume WSL? To use Windows/DOS commands from WSL, you will need to do something like cmd.exe /c "echo hello" see the interop docs.

It appears to work the same for the mingw64 git bash prompt, but I would certainly recommend WSL, instead.

you could also use start like this: cmd.exe /c start "notepad.exe"

I was running the commands using the console which comes with git (git-bash I think?)

does cmd.exe work for you from your git bash prompt?

I tried running cmd.exe in the prompt and got the following;

C:\Users\tim\conda-env-make\test>dir env
 Volume in drive C has no label.
 Volume Serial Number is BEFC-9299

 Directory of C:\Users\tim\conda-env-make\test\env

06/17/2020  08:04 AM    <DIR>          .
06/17/2020  08:04 AM    <DIR>          ..
06/17/2020  08:05 AM    <DIR>          downloads
               0 File(s)              0 bytes
               3 Dir(s)  255,711,010,816 bytes free

C:\Users\tim\conda-env-make\test> start "" /WAIT env/downloads/Miniconda3-latest-Windows-x86_64.exe /InstallationType=JustMe /AddToPath=0 /RegisterPython=0 /NoRegister=1 /NoS
cripts=1 /S /D=.\env\conda

C:\Users\tim\conda-env-make\test>dir env
 Volume in drive C has no label.
 Volume Serial Number is BEFC-9299

 Directory of C:\Users\tim\conda-env-make\test\env

06/17/2020  08:04 AM    <DIR>          .
06/17/2020  08:04 AM    <DIR>          ..
06/17/2020  08:05 AM    <DIR>          downloads
               0 File(s)              0 bytes
               3 Dir(s)  255,712,911,360 bytes free

C:\Users\tim\conda-env-make\test>

So at least wait seem to behave when I did that?

It looks like I have to give /D an absolute path?

start "" /WAIT env/downloads/Miniconda3-latest-Windows-x86_64.exe /InstallationType=JustMe /AddToPath=0 /RegisterPython=0 /NoRegister=1 /NoScripts=1 /S /D=C:\Users\tim\conda-env-make\test\env\conda

Looks like it might be working...

It is also unclear to me what I need to do in the Makefile here;
https://github.com/SymbiFlow/conda-env-make/pull/4/files#diff-b9ff15badc3b16747d79b9648d2ffafaR116-R118

GitHub

Number of improvements by mithro · Pull Request #4 · SymbiFlow/conda-env-make

Adding a test Running the test on travis

I'm confused. Is the issue that the Conda installer is popping up a prompt that is never getting acknowledged due to it running in CI?

I thought Conda generally used .bat files rather than Makefiles. And, as you've seen, cmd.exe is very limited in terms of what it can do.

You might try looking into Powershell, if you bump up against the limitations of what cmd.exe is capable of. Travis seems to start out in Powershell, and then drops down to sh or cmd depending on what you run...

I would certainly encourage the use of batch files (or powershell) to make things more clear - and more importantly, to be able to use quotes for potentially embedded space paths as well as handle multiple Windows commands more gracefully.

for the example, above:

tim@tim-win-test MINGW64 ~/conda-env-make/test (master)
$ start "" /WAIT env/downloads/Miniconda3-latest-Windows-x86_64.exe /S /D=C:\temp2
The system cannot find the file C:/Program Files/Git/WAIT.

there are several issues. for one, Windows expects backslashes \ at the commandline for directory path delimiters. Yes, some things work with either, but for instance you can't do something like dir env/ at a DOS prompt.

I'd consider putting the command(s) you want to call from MINGW64 or WSL in a batch file:

:: this is a sample file called mytask.bat
env\downloads\Miniconda3-latest-Windows-x86_64.exe /S /D=C:\temp2

:: this exit will allow a return to calling environment (otherwise you'd stay at the DOS command prompt when using start)
exit

to start a new process from DOS in a different window, it would be called like this:

C:\> start "my task description" /WAIT mytask.bat

or like this from MINGW64 or WSL to spawn a new process (no output) then exit:

$ cmd.exe /c  start "my task description" /WAIT mytask.bat

or like this to run the batch file in the current process, send everything to the screen, and return

$ cmd.exe /c start "my task description" /WAIT /B mytask.bat

that specific Miniconda line, if you really wanted to keep it with no batch files:

$ cmd.exe start "" /WAIT env\downloads\Miniconda3-latest-Windows-x86_64.exe /S /D=C:\temp2

the /S /D=C:\temp2 are assumed to be parameters for \Miniconda3-latest-Windows-x86_64.exe

should instead be something like:

$ cmd.exe /c start "running Miniconda3" /WAIT "env\downloads\Miniconda3-latest-Windows-x86_64.exe" /S /D=C:\temp2

Note that probably the only purpose of start I would think... is to start a separate process and let it run asynchronously. If the intent is just to run the command, then we can skip the start:

cmd.exe /c mytask.bat

Note that simple dos commands can be used like this: cmd.exe /c dir

or for your Miniconda3, something like this:

$ cmd.exe /c "env\downloads\Miniconda3-latest-Windows-x86_64.exe" /S /D=C:\temp2

does this help?

edit: a couple of other thoughts: you can call WSL from DOS like this: wsl.exe ls. You can also call powershell.exe from WSL or DOS. It can get quite mind bending.

I think all the confusion that came about is the fact that start works from the MINGW64 but not WSL. I'd suggest using cmd.exe to allow better interoperability with WSL.

I'm confused. Is the issue that the Conda installer is popping up a prompt that is never getting acknowledged due to it running in CI?

That is what I was trying to debug but I only have limited windows computer access.

I thought Conda generally used .bat files rather than Makefiles. And, as you've seen, cmd.exe is very limited in terms of what it can do.

The goal here is to have conda installed through make dependencies on Windows in a similar manner to Linux + Mac OS X.

You might try looking into Powershell, if you bump up against the limitations of what cmd.exe is capable of. Travis seems to start out in Powershell, and then drops down to sh or cmd depending on what you run...

Is powershell always available in Windows?

I would certainly encourage the use of batch files (or powershell) to make things more clear - and more importantly, to be able to use quotes for potentially embedded space paths as well as handle multiple Windows commands more gracefully.

The goal is to have all this run as part of the Makefile. The Makefile already disallows spaces in the install path for deliberate reason :-)

Note that probably the only purpose of start I would think... is to start a separate process and let it run asynchronously.

Actually the point of start is to wait until the process finishes. You can't continue to the next step until conda has finished installing.

Running cmd.exe inside MINGW64 just seems to start cmd.exe no matter what arguments I give cmd.exe?

tim@tim-win-test MINGW64 ~/conda-env-make/test (master)
$ cmd.exe /C dir
Microsoft Windows [Version 10.0.17763.1282]
(c) 2018 Microsoft Corporation. All rights reserved.

C:\Users\tim\conda-env-make\test>

It very much seems like arguments are not getting passed from the MINGW64 shell to the windows programs?

Well, I'm further along now - the makefile runs to the end but for some reason the downloaded Miniconda3 file is always newer than the python.exe file

     Prerequisite 'C:\Users\tim\conda-env-make\test\env\downloads\Miniconda3-latest-Windows-x86_64.exe' is newer than target 'C:\Users\tim\conda-env-make\test\env\conda\python.exe'.

The suggestion of copy /B %1+,, %1 on https://superuser.com/questions/10426/windows-equivalent-of-the-linux-command-touch doesn't seem to work...

Super User

Windows equivalent of the Linux command 'touch'?

What do you use when you want to update the date-modified field of a file on Windows? commands accessible via C++, .NET, C#, or something native to Windows (Vista preferably) tools/applications

I'm not familiar with this build process, but what exactly do you need to do?

I think many of your problems will go away if you decide to use Powershell, which is always available.

Additionally, you can prototype some things by installing Powershell Core for Linux.

For example, the command to create a new file (if that's what you're using touch to do) is:

New-Item -ItemType File newfile.txt

Or if you're using touch to update the last modified timestamp, you would run:

(Get-ChildItem newfile.txt).LastWriteTime = Get-Date

Additionally, cmd.exe really really cares about using \, but powershell will happily let you use either.

@gojimmypi - Make depends on the datetime of the files, hence I need to update the datetime of a file after updating it. On Linux you use the touch command to do that.

Here's a powershell function that reimplements touch like on posix:

function touch($fn) {
    if (Test-Path $fn) {
        (Get-ChildItem $fn).LastWriteTime = Get-Date
    } else {
        New-Item -Path $fn -ItemType File
    }
}

curious, as mingw32 cmd.exe /c dir returns a directory listing for me.

there does not seem to be a windows touch command, but in ming32, there is.

is the makefile running in Windows?

@xobs - How do I replace touch.cmd here -> https://github.com/SymbiFlow/conda-env-make/blob/1062560d3cf0c01244715991cf1c1e83a742ee99/touch.cmd with the powershell equivalent?

GitHub

SymbiFlow/conda-env-make

Repository containing common Makefiles for setting up conda environments. - SymbiFlow/conda-env-make

@xobs - I came up with https://github.com/mithro/conda-env-make/blob/master/touch.ps1 but I need to call it with powershell touch.ps1 <file> compared to the touch.cmd where I can just touch <file>?

GitHub

mithro/conda-env-make

Repository containing common Makefiles for setting up conda environments. - mithro/conda-env-make

@mithro Are you calling touch.cmd or touch (i.e. without the extension)? You could try just creating a file called touch.ps1 containing the following:

param (
    [string]$filename
)

if (Test-Path $filename) {
    (Get-ChildItem $filename).LastWriteTime = Get-Date
} else {
    New-Item -Path $filename -ItemType File | Out-Null
}

If that doesn't work, you may need to rename touch.cmd to something else, because I'm not sure the order of precedence of .cmd vs .ps1.

No, it does seem as though cmd.exe is too dumb to know that it needs to invoke powershell... And then you get all the weird escaping fun with spaces.

Hmm...

you have ming32 installed? how about: bash.exe -c "touch log.txt"

Can you avoid using cmd.exe entirely by setting SHELL := powershell -NoProfile at the top?

btw - powershell is invoked with powershell.exe, not cmd.exe, no?

@gojimmypi - I'm trying to do this only with stuff installed by default in Windows.

Once conda is installed, I have the full power of Python to do anything, until then I'm trying to have no dependencies.

how about powershell.exe (Get-ChildItem "log.txt").LastWriteTime = Get-Date

@gojimmypi - @xobs suggested that. I tried putting it into a touch.ps1 with the hope that cmd.exe would execute it when it was called touch (like I was trying with touch.cmd) but it appears that doesn't work.

I have resorted to just calling it with powershell in-front of it....

If you set the shell to powershell, things get a lot easier. For example, you can do away with the $(SEP) in your scripts.

Let me take a look at it and see what I get...

call it with powershell.exe ?

@xobs - You mean in the conda.mk script?

Current version which seems to mostly work is at https://github.com/mithro/conda-env-make/blob/master/conda.mk

GitHub

mithro/conda-env-make

Repository containing common Makefiles for setting up conda environments. - mithro/conda-env-make

OS detection stuff is at https://github.com/mithro/conda-env-make/blob/master/os.mk

GitHub

mithro/conda-env-make

Repository containing common Makefiles for setting up conda environments. - mithro/conda-env-make

if you have this in touch.ps1 (from above)

param (
    [string]$filename
)

if (Test-Path $filename) {
    (Get-ChildItem $filename).LastWriteTime = Get-Date
} else {
    New-Item -Path $filename -ItemType File | Out-Null
}

call it like this:

powershell.exe -file touch.ps1 log.txt

I can confirm that you need to change /NoRegister=1 -> /NoRegistry=1.

As of right now, it currently seems to work:

[1:51:33 PM] D:/Code/conda-env-make/test> make test-command
OS_TYPE=Windows CPU_TYPE=x86_64
'name: conda-env' 'conda-env'
[...]
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate conda-env
#
# To deactivate an active environment, use
#
#     $ conda deactivate

powershell D:\Code\conda-env-make\test\third_party\conda-env-make\touch.ps1 "D:\Code\conda-env-make\test\env\conda\envs\conda-env\python.exe"
call D:\Code\conda-env-make\test\env\conda\condabin\conda.bat activate conda-env && echo "Python is $(which python)"
"Python is $(which python)"
call D:\Code\conda-env-make\test\env\conda\condabin\conda.bat activate conda-env && python --version
Python 3.8.3
[1:53:56 PM] D:/Code/conda-env-make/test> make test-command
OS_TYPE=Windows CPU_TYPE=x86_64
'name: conda-env' 'conda-env'
call D:\Code\conda-env-make\test\env\conda\condabin\conda.bat activate conda-env && echo "Python is $(which python)"
"Python is $(which python)"
call D:\Code\conda-env-make\test\env\conda\condabin\conda.bat activate conda-env && python --version
Python 3.8.3
[1:54:31 PM] D:/Code/conda-env-make/test>

I have a few questions:

1. Will you assume wget exists on the Windows host, or would you like me to reimplement it in terms of either the PowerShell command Invoke-WebRequest or curl (which now ships as part of Windows)?
2. Would you like me to make the /NoRegistry=1 patch as a PR, or would you like to make that change yourself?
3. Is there anything more that needs to be done that isn't working in the above command?

@xobs - Did you change the Makefile to use shell := powershell at all or just test my latest version?

1. If we are assuming powershell, then we should have a wget replacement.
2. Yes
3. Maybe?

I'm just testing the latest solution.

make behaves very strangely. One idea I had was to set the SHELL such that it sources a profile that gives us the necessary calls, however make doesn't seem to like doing that.

So I think that your current approach -- replace calls to commands with calls to shims that explicitly call powershell -- is correct.

It seems to work for me, and with that PR it no longer creates registry entries. However it still creates Start Menu icons, and it shouldn't do that.

It looks like it's been an issue for at least two years: ContinuumIO/anaconda-issues#9729