Bash script for running a python script in each subdirectory
This shell script which is developed to batch process simulation results, runs a python script in each subdirectory of a parent folder.
storeDir="processedData"
pyFile="dataExtract.py"
if [ -d "$storeDir" ]; then
rm -Rf "$storeDir"
fi
for dir in A*/; do
cp "$pyFile" "$dir"
cd "$dir"
python3 "$pyFile"
rm "$pyFile"
cd ..
done
mkdir "$storeDir"
mv A*.dat "$storeDir"
echo "Files are moved to $storeDir."
echo "Done..."
Batch script for running a phython script in each subdirectory
This script runs a python script in each subdirectory of a parent folder. This code is developed to batch process simulation results without any commercial software. A tiny step towards building open source tools.
@echo off
set back=%cd%
for /d %%i in (%~dp0\*) do (
xcopy "dataExtract.py" "%%i"
cd "%%i"
echo current directory:
cd
python3 dataExtract.py
)
cd %back%
for /d %%i in (%~dp0\*) do (
cd "%%i"
del dataExtract.py
)
cd %back%
pause
Python script for extracting data from ascii files
This script reads ascii files, collects desired data, process and store processed data to a .dat (actually a csv) file. This can also be done using grep in linux terminal.Now, let’s jump into the script. Necessary packages like pandas, natsort, pathlib are imported first.
import os
import pandas as pd
from pathlib import Path
import natsort
Changing the working directory to the directory in which the file is located such that input files from this directory can be accessed.
os.chdir(os.path.dirname(os.path.abspath(__file__)))
CURR_DIR = os.getcwd()
Finding the directory name (not the full path) and listing the input files.
dmnFileName = CURR_DIR.split('\\')[-1]
temp_files = Path(CURR_DIR).rglob('*.int') #reads .int files (input files) only
files = [x for x in temp_files]
Making the skeleton.
values = []
# for A = 2.0
s2p0 = [a list of 33 numbers] % hidden data
# for A = 1.5
s1p5 = [a list of 33 numbers] % hidden data
# for A = 1.0
s1p0 = [a list of 33 numbers] % hidden data
# for A = 0.5
s0p5 = [a list of 33 numbers] % hidden data
n = float(dmnFileName.split('_')[1])
if n == 0.5:
s = s0p5
elif n == 1.0:
s = s1p0
elif n == 1.5:
s = s1p5
elif n == 2.0:
s = s2p0
Reading each file, collecting information, processing and storing for future use.
for name in natsort.os_sorted(files):
linesInHeader = 5
file = open(name,'r')
lines = file.readlines()[linesInHeader:]
lines = [line.strip('\n') for line in lines]
lines = [line.split(' ') for line in lines]
integral = float(lines[-1][-1])
values.append(integral)
file.close()
values = [val * 1000 for val in values]
try:
data1 = {"s": s[0:17],
"value": values[0:17]}
df1 = pd.DataFrame(data1)
print('\n', dmnFileName)
print('\nIncreasing\n____________________')
print(df1)
df1.to_csv(dmnFileName + '-increasing.dat', index = False)
data2 = {"s": s[16:33],
"value": values[16:33]}
df2 = pd.DataFrame(data2)
print('\nDecreasing\n____________________')
print(df2)
df2.to_csv(dmnFileName + '-decreasing.dat', index = False)
print("\n______Completed______\n")
except:
print("\nPLEASE CHECK IF THE TOTAL NUMBERS OF .int FILES is 33.\n")
#input("\nHit Enter to return....\n")
Python script for making PDF of exam routine and sending it to respective faculty member
This script reads exam schedule from an Excel file and creates a pdf that is sent to the respective faculty member. This script helped me a lot while performing as an exam coordinator in Sonargaon University (SU).
#===================================================================#
# Title: PDF Maker for faculties #
# Author: Dhiman Roy, Dept. of ME (BUET) #
# Date: April 04, 2021 #
# Licensed under Creative Commons #
#===================================================================#
import os
os.chdir(os.path.dirname(os.path.abspath(__file__)))
import pandas as pd
from fpdf import FPDF
myFile = "routine.xlsx"
data = pd.read_excel(myFile, usecols = ["Sort By","Date","Day","Slot","ERP Section Name",
"Course Code","Course Title","Teacher","Designation","Class Schedule"], header = 2)
teachers = data["Teacher"].unique().tolist()
print(teachers)
def makePdf(df, teacher):
titles = df.columns.tolist()
#print(titles )
dfData = df.values.tolist()
dfData.insert(0, titles)
#print(dfData[0])
myPdf = FPDF()
myPdf.add_page("L")
myPdf.set_font("Times", size = 16)
myPdf.cell(0, 10, txt = "Routine for Final Assessment, Spring 21", ln = 1, align = "C")
myPdf.cell(0, 10, txt = "Department of Mechanical Engineering", ln = 1, align = "C")
myPdf.cell(0, 10, txt = "Sonargaon University (SU)", ln = 1, align = "C")
myPdf.set_font("Times", size = 10)
myPdf.cell(0, 10, txt = "", ln = 1)
myPdf.set_font("Times", "I",size = 12)
myPdf.cell(0, 10, txt = "Teacher: " + teacher, align = "L")
myPdf.cell(0, 10, txt = "Designation: " + df["Designation"].tolist()[0], ln = 1, align = "R")
i=3
for dat in dfData:
if i == 3:
myPdf.set_font("Times", "B",size = 12)
else:
myPdf.set_font("Times", "",size = 10)
myPdf.cell(25, 10, txt = dat[0], border = 1, align = "C")
myPdf.cell(25, 10, txt = dat[1], border = 1, align = "C")
myPdf.cell(35, 10, txt = dat[2], border = 1, align = "C")
myPdf.cell(25, 10, txt = dat[4], border = 1, align = "C")
myPdf.cell(120, 10, txt = dat[5], border = 1, align = "C")
myPdf.cell(45, 10, txt = dat[3], border = 1, align = "C", ln = 1)
i = i + 1
myPdf.output(teacher + ".pdf")
print(teacher + ".pdf is created.")
i = 1
for tea in teachers:
dmnData = data.loc[data.Teacher == tea].sort_values(by = "Sort By") #filters data using teachers list
del dmnData["Sort By"]
dmnData["Date"] = dmnData["Date"].dt.strftime("%B %d, %Y") #Converts date to string
#print(dmnData)
makePdf(dmnData, tea)
i = i + 1
print("\n*******\nTotal %d files are saved.\n********\n" %(i-1))
input("Hit Enter to exit...")
The following script can send the generated pdf files to the respective faculty member.
#===================================================================#
# Title: Email Sender #
# Author: Dhiman Roy, Dept. of ME (BUET) #
# Date: April 05, 2021 #
# Licensed under Creative Commons #
#===================================================================#
import os
os.chdir(os.path.dirname(os.path.abspath(__file__)))
import pandas as pd
import smtplib
from email.message import EmailMessage
from email.header import Header
from email.utils import formataddr
import glob
files = glob.glob("*.pdf") #reads pdf files only
#print(files)
log = open("log.txt","w")
log.close()
senderName = "Exam Coordination Team, Dept. of ME, SU"
senderEmail = "emailaddress@gmail.com"
pw = input("Password: ")
with open("index.html") as htmlFile:
htmlCode = htmlFile.read()
htmlCode = htmlCode.split("Faculties")
def sendEmail(i, tea, mailList):
msg = EmailMessage()
msg["Subject"] = "Revised Routine for Final Assessment of Spring 2021"
msg["From"] = formataddr((str(Header(senderName , 'utf-8')), senderEmail))
for mail in mailList:
if tea in mail[0]:
mailAdd = mail[1]
msg["To"] = mailAdd
#print(mailAdd)
'''
with open("messageBody.txt") as textFile:
text = textFile.read()
msg.set_content(text)
'''
newHtmlCode = htmlCode[0] + tea + htmlCode[1]
msg.add_alternative(f"""\n\n{newHtmlCode}""", subtype = "html")
with open(tea + "-revised.pdf","rb") as attachment: #pdf files are needed to be ready
fileData = attachment.read()
fileName = attachment.name
msg.add_attachment(fileData, maintype = "application", subtype = ".pdf", filename = fileName)
'''
try:
with open(tea + ".pdf","rb") as attachment:
fileData = attachment.read()
fileName = attachment.name
msg.add_attachment(fileData, maintype = "application", subtype = ".pdf", filename = fileName)
except IOError:
print("%s.pdf is not found!!!" %tea)
'''
with smtplib.SMTP_SSL("smtp.gmail.com", 465) as server:
server.login(senderEmail,pw)
server.send_message(msg)
print("%02d. Successfully sent %s.pdf to %s" %(i, tea, mailAdd))
log.writelines("%02d. Successfully sent %s.pdf to %s\n" %(i, tea, mailAdd))
################## Edit this
classRoutine = "ClassRoutine.xlsx" #get mail ids from
##################
routine_data = pd.read_excel(classRoutine, usecols = ["Teacher","Email"], header = 6)
allData = routine_data.values.tolist()
################## edit this
myFile = "routine.xlsx" #gets exam routine from
##################
data = pd.read_excel(myFile, usecols = ["Teacher"], header = 2)
teachers = data["Teacher"].unique().tolist()
#print(teachers)
listOfMails=[]
teachermails = []
try:
for teacher in allData:
if teacher[0] in teachers:
if teacher[0] not in teachermails:
teachermails.append(teacher[0])
listOfMails.append([teacher[0], teacher[1]])
except IOError:
print("listOfMails not complete.")
#print(teachermails)
#print(allData)
#print(listOfMails)
up_teachers = ["Mr. X", "Dr. Y"]
print("Sending emails. Please wait.")
i = 1
for everyone in up_teachers:
log = open("log.txt","a")
try:
sendEmail(i, everyone, listOfMails)
except IOError:
print("%02d. sendEmail did not run for %s" %(i, everyone))
log.writelines("%02d. sendEmail did not run for %s\n" %(i, everyone))
i = i + 1
log.close()
input("Hit Return to exit...")
Bash codes for custom commands
alias goto='/mnt/c/Program\ Files/Opera/launcher.exe'
#alias search='goto www.google.com/search?q='
g() {
search=""
echo "Googling: $@"
for term in $@; do
search="$search%20$term"
done
search=${search/\%20/}
goto http://www.google.com/search?q=$search
}
y(){
sea=""
echo "Searching in Youtube: $@"
for term in $@; do
sea="$sea+$term"
done
sea=${sea/\+/}
goto https://www.youtube.com/results?search_query=$sea
}
alias km='/mnt/c/Program\ Files/KMPlayer/KMPlayer.exe'
op(){
nil=""
dmn=""
sl="/"
file="file:///"
col=":/"
loc=$(pwd)
link=${loc/\/mnt\//$nil}
base=$(removebackslash $@)
for term in $link$sl$base; do
dmn="$dmn%20$term"
done
dmn=${dmn/\%20/$nil}
dmn=${dmn/\//$col}
dmn=$file$dmn
echo "$dmn"
echo "Opening $@ in Opera"
goto $dmn
}
removebackslash(){
bs="\\"
str=${@//$bs/}
echo "$str"
}
Scheme code for creating points and monitors
;==========================================
; Title: Points and Monitors Generator
; Author: Dhiman Roy, ME-14, BUET
; Date: February 07, 2021
; Licensed under Creative Commons.
;==========================================
(newline)
(display "Generating points and creating monitors...")(newline)
(newline)
(define avg "Area-Weighted Average")
(define x-cor (list of x co-ordinates)) ;hidden data
(define y-cor (list of y co-ordinates)) ;hidden data
(
Do ((i 1 (+ i 1))) ((> i (length x-cor)))
(Ti-menu-load-string (format #f "surface/point-surface point_~a_~a ~a ~a" (* 10000 (list-ref x-cor (- i 1))) (* 10000 (list-ref y-cor (- i 1))) (list-ref x-cor (- i 1)) (list-ref y-cor (- i 1))))
(newline)
(Ti-menu-load-string (format #f "solve/monitors/surface/set-monitor pressure-monitor-~a avg pressure point_~a_~a () no no yes pres_~a_~a.txt 1 yes flow-time" i (* 10000 (list-ref x-cor (- i 1))) (* 10000 (list-ref y-cor (- i 1))) (* 10000 (list-ref x-cor (- i 1))) (* 10000 (list-ref y-cor (- i 1)))))
(newline)
(Ti-menu-load-string (format #f "solve/monitors/surface/set-monitor temperature-monitor-~a avg temperature point_~a_~a () no no yes temp_~a_~a.txt 1 yes flow-time" i (* 10000 (list-ref x-cor (- i 1))) (* 10000 (list-ref y-cor (- i 1))) (* 10000 (list-ref x-cor (- i 1))) (* 10000 (list-ref y-cor (- i 1)))))
(newline)
;(Ti-menu-load-string (format #f "solve/monitors/surface/set-monitor x-velocity-monitor-~a avg x-velocity point_~a_~a () no no yes xvel_~a_~a.txt 1 yes flow-time" i (* 10000 (list-ref x-cor (- i 1))) (* 10000 (list-ref y-cor (- i 1))) (* 10000 (list-ref x-cor (- i 1))) (* 10000 (list-ref y-cor (- i 1)))))
(newline)
(Ti-menu-load-string (format #f "solve/monitors/surface/set-monitor y-velocity-monitor-~a avg y-velocity point_~a_~a () no no yes yvel_~a_~a.txt 1 yes flow-time" i (* 10000 (list-ref x-cor (- i 1))) (* 10000 (list-ref y-cor (- i 1))) (* 10000 (list-ref x-cor (- i 1))) (* 10000 (list-ref y-cor (- i 1)))))
)
(newline)
(newline)
(
Do ((i 1 (+ i 1))) ((> i 51))
(display "*")
)
(newline)
(display "Points are generated.")(newline)
(display "Monitors are created..")(newline)
(display "Files will be saved in current working directory...")(newline)
(
Do ((i 1 (+ i 1))) ((> i 51))
(display "*")
)
(newline)
(newline)
Script for decoding Particle History Data (.xml) files (Exported from Ansys FLUENT)
This script can read DPM particle history data (.xml files) exported from Ansys FLUENT. It calculates DPM concentration inside a sphere at different times and writes them in a .dat file. A part of this script has been used to process data for this article.
For the script, please email me.
Demonstration:
Python script for calculating average drag coefficient
#==========================================
# Title: Average drag coefficient calculator
# Author: Dhiman Roy, ME-14, BUET
# Date: March 28, 2021
# Licensed under Creative Commons.
#==========================================
import os
os.chdir(os.path.dirname(os.path.abspath(__file__)))
pardir = os.path.abspath(os.path.join(__file__, os.pardir))
resFilePath = "G:\Sumlation Results (Dhiman)"
def askToChange(parameter, val):
print("%s is set to %f" %(parameter, val))
prompt = input("Do you want to change? (Y/N): ")
if prompt == 'y' or prompt == 'Y':
val = input("Enter new value for %s: " %parameter)
elif prompt == 'n' or prompt == 'N':
val = val
else:
val = val
#askToChange("Frontal Area", frontalArea)
return float(val)
try:
#Edit
getValueFrom = 1000 #Start Claculation from
getValueTo = 5000 #Upto which iteration
linesInHeader = 2 #number of lines as header
frontalArea = 0.74882341 #0.83443312 FOR SHIELD
frontalArea = askToChange("Frontal Area", frontalArea)
density = 1.225
velocity = float(input("Enter velocity (m/s): "))
#Think_before_making_any_change
linesToSkip = getValueFrom + linesInHeader - 1
file = open("cd-biker", "r")
contents = file.readlines()[linesToSkip:(getValueTo + linesInHeader)]
iter = []
cd_each_iter = []
#simpson = []
i = 0
sum = 0
for line in contents:
splitline = line.split(" ")
iter.append(float(splitline[0]))
cd_each_iter.append(float(splitline[1]))
#print(len(iter))
while i<len(iter) - 2:
integral = (1 / 3) * (cd_each_iter[i] + 4 * cd_each_iter[i+1] + cd_each_iter[i+2])
sum = sum + integral
#simpson.append(integral)
i = i + 2
avg_cd = sum / (iter[len(iter) - 1] - iter[0])
dragForce = 0.5 * avg_cd * density * frontalArea * velocity * velocity
#print("Sum is", sum)
#print(iter)
#print(cd_each_iter)
#print(simpson)
#print(sum)
msg = "".join(["Average drag coefficient (from iteration ", str(getValueFrom), " to iteration ", str(getValueTo),") is ", str(avg_cd), " and drag on bike is ", str(dragForce), " N\n"])
#print("Average drag coefficient from (iteration ", getValueFrom, " to iteration ", getValueTo,") is ", avg_cd,sep="")
print("\n******************************************************************************\n", msg)
try:
savefile = open("avgerage_cd.txt", "a")
savefile.write(msg)
print(" Data stored!!\n******************************************************************************\n")
except IOError:
print("Couldn't open file to save!")
finally:
if "savefile" in locals():
savefile.close()
except IOError:
print("Couldn't open file!")
finally:
if "file" in locals():
file.close()
def writeAllRes():
try:
res = "".join([pardir, ",", str(avg_cd), ",", str(dragForce), "\n"])
allResultToFile = open(resFilePath + "\All-Simulation_Result.csv", "a")
allResultToFile.write(res)
print('Allres saved')
except IOError:
print("Couldn't open 'All-Simulation_Result.csv' to store data!")
finally:
if "allResultToFile" in locals():
allResultToFile.close()
try:
os.mkdir(resFilePath)
print('Directory ')
writeAllRes()
except:
writeAllRes()
input("Hit Enter to exit..........\n")
Python script for creating Java Script that populates a webform
This script reads the obtained marks of the students from an Excel file and creates a Java Script that can be used to populate a certain webform automatically. In short, data entry is not a problem anymore.
#===================================================================#
# Title: JavaScript Maker for Marks input #
# Author: Dhiman Roy, Dept. of ME (BUET) #
# Date: April 12, 2021 #
# Licensed under Creative Commons #
#===================================================================#
import os
import pandas as pd
from pathlib import Path
os.chdir(os.path.dirname(os.path.abspath(__file__)))
try:
os.mkdir("JS")
print("\n\tJS directory is created.")
except IOError:
print("\n\tJS directory already exists.")
CURR_DIR = os.getcwd()
jsDir = os.path.join(CURR_DIR, "JS")
#print(CURR_DIR)
txt_folder = Path(CURR_DIR).rglob('*.xlsx') #reads .xlsx files only
#print(txt_folder)
files = [x for x in txt_folder]
#print(files)
#print("\n")
def midOrall():
value = input("\n\tDo you want script for Mid (m) or All (a)? ")
if value == 'm' or value == 'a':
#print(value)
return value
else:
print("\n\tPlease select one...")
midOrall()
def quitOrNot():
quitDmn = input("\t\tDo you want to quit? (Y/N): ")
if quitDmn == 'y' or quitDmn == 'Y' or quitDmn == 'yes' or quitDmn == 'Yes' or quitDmn == 'YES':
quit()
elif quitDmn == 'n' or quitDmn == 'N' or quitDmn == 'no' or quitDmn == 'No' or quitDmn == 'NO':
print()
else:
quitOrNot()
def mustQuit():
input("\t\tHit enter to exit.")
quit()
def stringconv(dmnMark, dmnRow, dmnSec, dmnFile, filename):
try:
return str(int(dmnMark))
except:
print("\t\tEmpty cell in SL %i, Sheet name: %s, Workbook: %s" %(dmnRow + 1, dmnSec, dmnFile))
#quitOrNot()
file.close()
os.remove(filename)
mustQuit()
if midOrall() == 'm':
for name in sorted(files):
sheetForSection = pd.ExcelFile(name)
section = sheetForSection.sheet_names
fName = os.path.basename(name)
substring = ['theory', 'THEORY', 'Theory']
theoryfile = []
if substring[0] in os.path.splitext(os.path.basename(name))[0] or substring[1] in os.path.splitext(os.path.basename(name))[0] or substring[2] in os.path.splitext(os.path.basename(name))[0]:
header_val = 1
theoryfile.append(os.path.basename(name))
else:
header_val = 0
print("\n\tSections from %s" %fName)
for sec in section:
data = pd.read_excel(name, index_col = 0, usecols = "C:H", header = header_val, sheet_name = sec)
#print(data)
print("\t-%s" %sec)
marks = data.values.tolist()
#print(marks)
filename = sec + "_" + os.path.splitext(os.path.basename(name))[0] + ".txt"
#print(filename)
if header_val == 1:
file = open(os.path.join(jsDir, filename),"w")
i = 0
for each in marks:
if each[3] == 'abs':
file.write("document.getElementById('midterm_%d').value = '%s';\n" %(i, each[3])) #mid absent
else:
file.write("document.getElementById('midterm_%d').value = '%s';\n" %(i, stringconv(each[3], i, sec, fName, filename))) #mid
file.write("calTotal(midterm_%d);\n" %i)
i = i + 1
file.close()
else:
print("\n\t...Sessionals are skipped...")
else:
for name in sorted(files):
sheetForSection = pd.ExcelFile(name)
section = sheetForSection.sheet_names
fName = os.path.basename(name)
substring = ['theory', 'THEORY', 'Theory']
theoryfile = []
if substring[0] in os.path.splitext(os.path.basename(name))[0] or substring[1] in os.path.splitext(os.path.basename(name))[0] or substring[2] in os.path.splitext(os.path.basename(name))[0]:
header_val = 1
theoryfile.append(os.path.basename(name))
else:
header_val = 0
print("\n\tSections from %s" %fName)
for sec in section:
data = pd.read_excel(name, index_col = 0, usecols = "C:H", header = header_val, sheet_name = sec)
#print(data)
print("\t-%s" %sec)
marks = data.values.tolist()
#print(marks)
filename = sec + "_" + os.path.splitext(os.path.basename(name))[0] + ".txt"
#print(filename)
if header_val == 0:
file = open(os.path.join(jsDir, filename),"w")
i = 0
for each in marks:
file.write("document.getElementById('attendance_%d').value = '%s';\n" %(i, stringconv(each[0], i, sec, fName, filename))) #attendance
file.write("document.getElementById('performance_%d').value = '%s';\n" %(i, stringconv(each[1], i, sec, fName, filename))) #classtest
if each[2] == 'abs':
file.write("document.getElementById('sessional_%d').value = '%s';\n" %(i, each[2])) #sessional final absent
else:
file.write("document.getElementById('sessional_%d').value = '%s';\n" %(i, stringconv(each[2], i, sec, fName, filename))) #sessoinal final
file.write("calTotal(sessional_%d);\n" %i)
i = i + 1
file.close()
else:
file = open(filename,"w")
i = 0
for each in marks:
file.write("document.getElementById('attendance_%d').value = '%s';\n" %(i, stringconv(each[0], i, sec, fName, filename))) #attendance
file.write("document.getElementById('classtest_%d').value = '%s';\n" %(i, stringconv(each[1], i, sec, fName, filename))) #classtest
file.write("document.getElementById('assignment_%d').value = '%s';\n" %(i, stringconv(each[2], i, sec, fName, filename))) #assignment
if each[4] == 'abs':
file.write("document.getElementById('final_%d').value = '%s';\n" %(i, each[4])) #final absent
else:
file.write("document.getElementById('final_%d').value = '%s';\n" %(i, stringconv(each[4], i, sec, fName, filename))) #final
if each[3] == 'abs':
file.write("document.getElementById('midterm_%d').value = '%s';\n" %(i, each[3])) #mid absent
else:
file.write("document.getElementById('midterm_%d').value = '%s';\n" %(i, stringconv(each[3], i, sec, fName, filename))) #mid
file.write("calTotal(midterm_%d);\n" %i)
i = i + 1
file.close()
print("\n\t\t====================")
print("\t\tScripts are ready!!!")
print("\t\t====================")
#print("\n")
input("\n\t\tPress enter to exit...\n")
Script for Fast Fourier Transformation
Conversion of available experimental data from time domain to frequency domain.
Clearing workspace and command window-
clear;
clc;
Importing data.csv-
data = readtable("data.csv");
Spliting into vectors-
time = data{:,1};
V = data{:,2};
Determining dt, length of the vectors and sampling frequency-
l1 = length(V);
dt = (time(end)-time(1))/(l1-1);
fs = 1/dt;
FFT-
fft_V = fft(V, l1)*(2/l1);
abs_V = abs(fft_V);
freq = 0:(1/time(end)):fs/2-(1/time(end));
Ploting raw data (Given in time domain)-
plot(time, V)
xlabel('Time')
xlim([0 125])
ylabel('Heat Flux')
ylim([-10 50])
grid on
Ploting FFT-
figure
plot(freq, abs_V(1:length(freq)))
xlabel('Frequency')
ylim([-10 110])
xlim([-0.025 0.1])
grid on
